[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:44 dist-test-slave-0n27 krb5kdc[1835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:44 dist-test-slave-0n27 krb5kdc[1835](info): set up 2 sockets
Sep 28 14:40:44 dist-test-slave-0n27 krb5kdc[1835](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 14:40:45.818459 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:40:45 dist-test-slave-0n27 krb5kdc[1835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:40:45.929263 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:45539
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:45539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:46.025707 1848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:46.025928 1848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:46.025993 1848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:46.029731 1848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:46.029774 1848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:46.029846 1848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:46.029893 1848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:46.029928 1848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:46.034560 1848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:45539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:45539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:46.035408 1848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:46.036293 1848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:46.044170 1853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:46.044260 1848 server_base.cc:1034] running on GCE node
W20260928 14:40:46.044181 1856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:46.044272 1854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:46.044889 1848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:46.045859 1848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:46.047011 1848 hybrid_clock.cc:648] HybridClock initialized: now 1790606446046993 us; error 30 us; skew 500 ppm
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:46.049341 1848 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:40:46.050113 1848 webserver.cc:460] Webserver started at http://127.1.201.62:44051/ using document root <none> and password file <none>
I20260928 14:40:46.050529 1848 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:46.050580 1848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:46.050742 1848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:46.052364 1848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "f7e1d6e0915546258d7a1950706c762b"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "42d1e705b2170c59e045ee8f27e5eea2"
server_key_iv: "6c2e104763588c3983df98c1babe42e9"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.052821 1848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "f7e1d6e0915546258d7a1950706c762b"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "42d1e705b2170c59e045ee8f27e5eea2"
server_key_iv: "6c2e104763588c3983df98c1babe42e9"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.056108 1848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:40:46.058147 1863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:46.059170 1848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:46.059303 1848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "f7e1d6e0915546258d7a1950706c762b"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "42d1e705b2170c59e045ee8f27e5eea2"
server_key_iv: "6c2e104763588c3983df98c1babe42e9"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.059401 1848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:46.081581 1848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:46.084059 1848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:46.084254 1848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:46.090695 1848 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:45539
I20260928 14:40:46.090713 1915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:45539 every 8 connection(s)
I20260928 14:40:46.091459 1848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:46.093701 1916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:46.094882 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 1848
I20260928 14:40:46.095047 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:40:46.096567 1828 external_mini_cluster.cc:1428] Setting key 68fbcd2f983d2673ca6fc4a50dcfc488
I20260928 14:40:46.099964 1916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b: Bootstrap starting.
I20260928 14:40:46.102022 1916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:46.102646 1916 log.cc:824] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:46.104373 1916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b: No bootstrap required, opened a new log
I20260928 14:40:46.107800 1916 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e1d6e0915546258d7a1950706c762b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45539 } }
I20260928 14:40:46.108000 1916 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:46.108034 1916 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e1d6e0915546258d7a1950706c762b, State: Initialized, Role: FOLLOWER
I20260928 14:40:46.108407 1916 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e1d6e0915546258d7a1950706c762b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45539 } }
I20260928 14:40:46.108546 1916 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:46.108628 1916 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:46.108736 1916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:46.109452 1916 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e1d6e0915546258d7a1950706c762b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45539 } }
I20260928 14:40:46.109802 1916 leader_election.cc:302] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7e1d6e0915546258d7a1950706c762b; no voters:
I20260928 14:40:46.110062 1916 leader_election.cc:288] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:46.110160 1921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:46.110347 1921 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [term 1 LEADER]: Becoming Leader. State: Replica: f7e1d6e0915546258d7a1950706c762b, State: Running, Role: LEADER
I20260928 14:40:46.110672 1921 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e1d6e0915546258d7a1950706c762b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45539 } }
I20260928 14:40:46.111227 1916 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:46.112176 1922 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7e1d6e0915546258d7a1950706c762b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e1d6e0915546258d7a1950706c762b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45539 } } }
I20260928 14:40:46.112218 1923 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7e1d6e0915546258d7a1950706c762b. Latest consensus state: current_term: 1 leader_uuid: "f7e1d6e0915546258d7a1950706c762b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e1d6e0915546258d7a1950706c762b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45539 } } }
I20260928 14:40:46.112285 1922 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:46.112306 1923 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:46.112607 1925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:46.115731 1925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:46.120100 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:46.099906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.1.201.62:45539)
0928 14:40:46.100311 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:46.100330 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:46.101352 (+ 1022us) server_negotiation.cc:407] Connection header received
0928 14:40:46.104264 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:46.104280 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:46.104597 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:46.104909 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:46.106948 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.108275 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.109546 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.110184 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.110346 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:46.114969 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:46.114975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:46.114990 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:46.115020 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:46.117070 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.117666 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.117671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.117676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.117842 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.118168 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.118171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.118173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.118496 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:46.118698 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:46.119163 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 14:40:46.119387 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20260928 14:40:46.121208 1925 catalog_manager.cc:1347] Generated new cluster ID: fc60590625c6408a979684732c791d5c
I20260928 14:40:46.121258 1925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:46.140210 1925 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:46.141176 1925 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:46.145476 1925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7e1d6e0915546258d7a1950706c762b: Generated new TSK 0
I20260928 14:40:46.146129 1925 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:40:46.176357 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:45539
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:46.276961 1942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:46.277159 1942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:46.277194 1942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:46.280908 1942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:46.280953 1942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:46.281005 1942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:40:46.285456 1942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:45539
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:46.286314 1942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:46.287078 1942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:46.295284 1947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:46.295324 1948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:46.295535 1942 server_base.cc:1034] running on GCE node
W20260928 14:40:46.295621 1950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:46.296038 1942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:46.296701 1942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:46.297866 1942 hybrid_clock.cc:648] HybridClock initialized: now 1790606446297840 us; error 33 us; skew 500 ppm
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:46.300382 1942 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:40:46.301189 1942 webserver.cc:460] Webserver started at http://127.1.201.1:36071/ using document root <none> and password file <none>
I20260928 14:40:46.301630 1942 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:46.301678 1942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:46.301898 1942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:46.303560 1942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "b6e5f532d3b54811b7fc3d0f54461e8a"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "be94e0f2af807004cb0c85459d300eb5"
server_key_iv: "51b1bf258bc2fb4ce63093e8c93ecc26"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.303994 1942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "b6e5f532d3b54811b7fc3d0f54461e8a"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "be94e0f2af807004cb0c85459d300eb5"
server_key_iv: "51b1bf258bc2fb4ce63093e8c93ecc26"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.307386 1942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:40:46.309370 1957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:46.310461 1942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:46.310596 1942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "b6e5f532d3b54811b7fc3d0f54461e8a"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "be94e0f2af807004cb0c85459d300eb5"
server_key_iv: "51b1bf258bc2fb4ce63093e8c93ecc26"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.310724 1942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:46.356652 1942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:46.359516 1942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:46.359683 1942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:46.360157 1942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:40:46.360826 1942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:46.360895 1942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:46.360973 1942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:46.361022 1942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:46.368945 1942 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:35301
I20260928 14:40:46.368971 2070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:35301 every 8 connection(s)
I20260928 14:40:46.369681 1942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:40:46.371456 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 1942
I20260928 14:40:46.371551 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:40:46.371801 1828 external_mini_cluster.cc:1428] Setting key 94becad885aa5a2ee126af6fb71a249f
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
I20260928 14:40:46.382673 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:46.371233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:57027 (local address 127.1.201.62:45539)
0928 14:40:46.371360 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:46.371363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:46.372232 (+ 869us) server_negotiation.cc:407] Connection header received
0928 14:40:46.373364 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:46.373367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:46.373421 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:46.373502 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:46.374567 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.375044 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.376620 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.376838 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.376913 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:46.379332 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:46.379342 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:46.379344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:46.379366 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:46.380743 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.381200 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.381208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.381209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.381275 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.381595 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.381598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.381599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.381755 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:46.381902 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:46.382335 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 14:40:46.382446 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
I20260928 14:40:46.383339 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:46.371669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:45539 (local address 127.1.201.1:57027)
0928 14:40:46.372097 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:46.372131 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:46.373167 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:46.373664 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:46.373670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:46.374015 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:46.374420 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:46.374430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.375190 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.375195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:46.376501 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:46.376514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.376977 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.376983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:46.377184 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:46.377649 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:46.377670 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:46.379153 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:46.380848 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:46.380853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:46.380866 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:46.381095 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:46.381355 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:46.381358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:46.381360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:46.381502 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:46.381901 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:46.381907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:46.382111 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 14:40:46.382267 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 14:40:46.382545 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":130,"threads_started":1}
I20260928 14:40:46.384354 2071 heartbeater.cc:347] Connected to a master server at 127.1.201.62:45539
I20260928 14:40:46.384574 2071 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:46.384986 2071 heartbeater.cc:511] Master 127.1.201.62:45539 requested a full tablet report, sending...
I20260928 14:40:46.386472 1880 ts_manager.cc:194] Registered new tserver with Master: b6e5f532d3b54811b7fc3d0f54461e8a (127.1.201.1:35301)
I20260928 14:40:46.387552 1880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:57027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:40:46.396950 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:45539
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:46.499691 2076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:46.499899 2076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:46.499938 2076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:46.503868 2076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:46.503913 2076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:46.503966 2076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:40:46.508828 2076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:45539
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:46.509613 2076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:46.510509 2076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:40:46.519706 2076 server_base.cc:1034] running on GCE node
W20260928 14:40:46.519595 2082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:46.519665 2081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:46.519614 2084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:46.520365 2076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:46.520994 2076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:46.522161 2076 hybrid_clock.cc:648] HybridClock initialized: now 1790606446522140 us; error 45 us; skew 500 ppm
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:46.524745 2076 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:40:46.525571 2076 webserver.cc:460] Webserver started at http://127.1.201.2:40939/ using document root <none> and password file <none>
I20260928 14:40:46.526150 2076 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:46.526240 2076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:46.526438 2076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:46.528330 2076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "99f49ab9374d4a8f817cac1f7fb83c32"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "6e0ed62be5bd26e397b302eba425a374"
server_key_iv: "7b10952fcd5a4d963b305f096bc68173"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.528995 2076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "99f49ab9374d4a8f817cac1f7fb83c32"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "6e0ed62be5bd26e397b302eba425a374"
server_key_iv: "7b10952fcd5a4d963b305f096bc68173"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.532457 2076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 14:40:46.534730 2091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:46.535800 2076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:46.535925 2076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "99f49ab9374d4a8f817cac1f7fb83c32"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "6e0ed62be5bd26e397b302eba425a374"
server_key_iv: "7b10952fcd5a4d963b305f096bc68173"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.536029 2076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:46.559630 2076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:46.562667 2076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:46.562942 2076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:46.563906 2076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:40:46.565372 2076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:46.565476 2076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:46.565564 2076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:46.565600 2076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:46.581236 2204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:33505 every 8 connection(s)
I20260928 14:40:46.581272 2076 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:33505
I20260928 14:40:46.582854 2076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:40:46.591897 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 2076
I20260928 14:40:46.592021 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:40:46.592283 1828 external_mini_cluster.cc:1428] Setting key 4424fc01cf970cc9bd9928c18e0f895e
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:46.596802 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:46.585038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:47859 (local address 127.1.201.62:45539)
0928 14:40:46.585229 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:46.585234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:46.586254 (+ 1020us) server_negotiation.cc:407] Connection header received
0928 14:40:46.587662 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:46.587665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:46.587710 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:46.587789 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:46.588861 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.589365 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.590493 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.590723 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.590816 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:46.593171 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:46.593174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:46.593176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:46.593197 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:46.594588 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.595223 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.595230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.595233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.595331 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.595651 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.595658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.595659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.595890 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:46.595993 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:46.596442 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 14:40:46.596561 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 14:40:46.597348 2207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:46.585547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:45539 (local address 127.1.201.2:47859)
0928 14:40:46.586078 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:46.586141 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:46.587445 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:46.587944 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:46.587950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:46.588293 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:46.588732 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:46.588744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.589473 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.589476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:46.590381 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:46.590390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.590864 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.590867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:46.590994 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:46.591479 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:46.591494 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:46.593010 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:46.594703 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:46.594711 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:46.594728 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:46.595104 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:46.595428 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:46.595430 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:46.595431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:46.595554 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:46.595997 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:46.596002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:46.596207 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 14:40:46.596361 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 14:40:46.596571 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":112,"threads_started":1}
I20260928 14:40:46.598521 2205 heartbeater.cc:347] Connected to a master server at 127.1.201.62:45539
I20260928 14:40:46.598762 2205 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:46.599293 2205 heartbeater.cc:511] Master 127.1.201.62:45539 requested a full tablet report, sending...
I20260928 14:40:46.600310 1880 ts_manager.cc:194] Registered new tserver with Master: 99f49ab9374d4a8f817cac1f7fb83c32 (127.1.201.2:33505)
I20260928 14:40:46.600809 1880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:47859
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:40:46.618340 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:45539
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:46.717249 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:46.717449 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:46.717514 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:46.720939 2210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:46.720983 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:46.721032 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:40:46.725493 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:45539
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:46.726397 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:46.727214 2210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:46.735605 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:46.735680 2210 server_base.cc:1034] running on GCE node
W20260928 14:40:46.735652 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:46.735641 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:46.736153 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:46.736697 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:46.737841 2210 hybrid_clock.cc:648] HybridClock initialized: now 1790606446737818 us; error 35 us; skew 500 ppm
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:46.740213 2210 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:40:46.740931 2210 webserver.cc:460] Webserver started at http://127.1.201.3:35789/ using document root <none> and password file <none>
I20260928 14:40:46.741377 2210 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:46.741429 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:46.741567 2210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:46.743388 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "b8a4e0efd0f642408e4221584bce96c1"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "6b1f5c83e755dee001112901206e4e49"
server_key_iv: "83a16ea6ecf78404845fcd90e1cb8a95"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.743839 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "b8a4e0efd0f642408e4221584bce96c1"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "6b1f5c83e755dee001112901206e4e49"
server_key_iv: "83a16ea6ecf78404845fcd90e1cb8a95"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.747061 2210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:40:46.749155 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:46.750218 2210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:46.750335 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "b8a4e0efd0f642408e4221584bce96c1"
format_stamp: "Formatted at 2026-09-28 14:40:46 on dist-test-slave-0n27"
server_key: "6b1f5c83e755dee001112901206e4e49"
server_key_iv: "83a16ea6ecf78404845fcd90e1cb8a95"
server_key_version: "encryptionkey@0"
I20260928 14:40:46.750429 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:46.770635 2210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:46.772934 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:46.773176 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:46.773658 2210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:40:46.774382 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:46.774448 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:46.774500 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:46.774549 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:46.782878 2210 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:35765
I20260928 14:40:46.782902 2339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:35765 every 8 connection(s)
I20260928 14:40:46.783617 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:46.793519 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 2210
I20260928 14:40:46.793633 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:40:46.793920 1828 external_mini_cluster.cc:1428] Setting key 413576a9cd7ff4ca2b3b032b0a446463
I20260928 14:40:46.795083 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:46.785142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:51161 (local address 127.1.201.62:45539)
0928 14:40:46.785324 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:46.785329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:46.786059 (+ 730us) server_negotiation.cc:407] Connection header received
0928 14:40:46.787172 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:46.787175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:46.787219 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:46.787287 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:46.788181 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.788675 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.789769 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.790041 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.790130 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:46.792169 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:46.792172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:46.792175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:46.792204 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:46.793256 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.793717 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.793720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.793722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.793832 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.794135 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.794140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.794141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.794299 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:46.794395 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:46.794840 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 14:40:46.794939 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 14:40:46.795902 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:46.785487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:45539 (local address 127.1.201.3:51161)
0928 14:40:46.785933 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:46.785969 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:46.787008 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:46.787422 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:46.787427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:46.787704 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:46.788069 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:46.788080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.788847 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.788850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:46.789667 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:46.789675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.790164 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.790167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:46.790304 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:46.790798 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:46.790812 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:46.792025 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:46.793379 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:46.793384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:46.793395 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:46.793611 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:46.793927 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:46.793930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:46.793931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:46.794047 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:46.794393 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:46.794399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:46.794624 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 14:40:46.794788 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 14:40:46.795036 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
I20260928 14:40:46.796885 2340 heartbeater.cc:347] Connected to a master server at 127.1.201.62:45539
I20260928 14:40:46.797075 2340 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:46.797451 2340 heartbeater.cc:511] Master 127.1.201.62:45539 requested a full tablet report, sending...
I20260928 14:40:46.798316 1880 ts_manager.cc:194] Registered new tserver with Master: b8a4e0efd0f642408e4221584bce96c1 (127.1.201.3:35765)
I20260928 14:40:46.798763 1880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:51161
I20260928 14:40:46.807636 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:40:46.829912 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:46.822313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.1.201.1:35301)
0928 14:40:46.822584 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:46.822596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:46.822615 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:40:46.822700 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:46.822704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:46.822836 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:46.822932 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:46.823486 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.824128 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.824822 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.825114 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.825210 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:46.827280 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:46.827288 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:46.827297 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:46.827316 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:46.828533 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.828875 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.828879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.828880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.828961 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.829158 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.829160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.829161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.829337 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:46.829468 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:46.829592 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 14:40:46.829722 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
W20260928 14:40:46.830683 1985 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36458
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[1835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:40:46.852926 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:46.846581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.1.201.1:35301)
0928 14:40:46.846709 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:46.846712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:46.846777 (+ 65us) server_negotiation.cc:407] Connection header received
0928 14:40:46.846855 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:46.846857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:46.846888 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:46.846944 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:46.847443 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.847863 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.848539 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:46.848728 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:46.848793 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:46.850667 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:46.850669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:46.850671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:46.850689 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:46.851737 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.852050 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.852052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.852053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.852112 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:46.852314 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:46.852316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:46.852317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:46.852436 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:46.852519 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:46.852680 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 14:40:46.852755 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:40:46.853752 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 1942
I20260928 14:40:46.859254 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 2076
I20260928 14:40:46.866765 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 2210
I20260928 14:40:46.871443 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 1848
2026-09-28T14:40:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2018 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[2355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[2355](info): set up 2 sockets
Sep 28 14:40:46 dist-test-slave-0n27 krb5kdc[2355](info): commencing operation
krb5kdc: starting...
W20260928 14:40:47.818123 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:40:47 dist-test-slave-0n27 krb5kdc[2355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:40:47.926774 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:39785
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:39785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:48.022083 2368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:48.022316 2368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:48.022394 2368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:48.025774 2368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:48.025872 2368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:48.025913 2368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:48.025969 2368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:48.026010 2368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:48.030493 2368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:39785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:39785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:48.031320 2368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:48.032145 2368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:48.039260 2374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:48.039307 2368 server_base.cc:1034] running on GCE node
W20260928 14:40:48.039243 2376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:48.039243 2373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:48.039842 2368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:48.040716 2368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:48.041851 2368 hybrid_clock.cc:648] HybridClock initialized: now 1790606448041822 us; error 37 us; skew 500 ppm
Sep 28 14:40:48 dist-test-slave-0n27 krb5kdc[2355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:48.044210 2368 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:40:48.045017 2368 webserver.cc:460] Webserver started at http://127.1.201.62:40777/ using document root <none> and password file <none>
I20260928 14:40:48.045627 2368 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:48.045727 2368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:48.045969 2368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:48.047681 2368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "f2e971196bb96aa45eea10dee2ae0ef5"
server_key_iv: "7350b644d2ce2009e818bb8fe8322740"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.048182 2368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "f2e971196bb96aa45eea10dee2ae0ef5"
server_key_iv: "7350b644d2ce2009e818bb8fe8322740"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.051154 2368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 14:40:48.053169 2383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:48.054275 2368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:48.054397 2368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "f2e971196bb96aa45eea10dee2ae0ef5"
server_key_iv: "7350b644d2ce2009e818bb8fe8322740"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.054491 2368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:48.067247 2368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:48.069396 2368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:48.069564 2368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:48.075397 2368 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:39785
I20260928 14:40:48.075433 2436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:39785 every 8 connection(s)
I20260928 14:40:48.076212 2368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:48.078485 2437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:48.081643 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 2368
I20260928 14:40:48.081764 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:40:48.082064 1828 external_mini_cluster.cc:1428] Setting key d8c35b334193408e74c03af4c88424df
I20260928 14:40:48.083541 2437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431: Bootstrap starting.
I20260928 14:40:48.085652 2437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:48.086383 2437 log.cc:824] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:48.088095 2437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431: No bootstrap required, opened a new log
Sep 28 14:40:48 dist-test-slave-0n27 krb5kdc[2355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:48.091981 2437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39785 } }
I20260928 14:40:48.092272 2437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:48.092334 2437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7264e4f7cf4c9aaa45b0e6fd7a8431, State: Initialized, Role: FOLLOWER
I20260928 14:40:48.092792 2437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39785 } }
I20260928 14:40:48.092931 2437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:48.093014 2437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:48.093104 2437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:48.093894 2437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39785 } }
I20260928 14:40:48.094238 2437 leader_election.cc:302] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e7264e4f7cf4c9aaa45b0e6fd7a8431; no voters:
I20260928 14:40:48.094511 2437 leader_election.cc:288] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:48.094595 2442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:48.094765 2442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [term 1 LEADER]: Becoming Leader. State: Replica: 4e7264e4f7cf4c9aaa45b0e6fd7a8431, State: Running, Role: LEADER
I20260928 14:40:48.095009 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.083243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.1.201.62:39785)
0928 14:40:48.083582 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:48.083595 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:48.083623 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:40:48.084194 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:48.084208 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:48.084468 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:48.084705 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:48.085495 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.086547 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.087536 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.087914 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.088018 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.090497 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:48.090502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:48.090513 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:48.090536 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:48.092160 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:48.092662 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:48.092675 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:48.092679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:48.092785 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:48.093078 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:48.093082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:48.093083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:48.093479 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:48.093670 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:48.093957 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 14:40:48.094191 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":104,"threads_started":1}
I20260928 14:40:48.095028 2442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39785 } }
I20260928 14:40:48.095579 2437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:48.096093 2442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e7264e4f7cf4c9aaa45b0e6fd7a8431. Latest consensus state: current_term: 1 leader_uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39785 } } }
I20260928 14:40:48.096215 2442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:48.097141 2443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7264e4f7cf4c9aaa45b0e6fd7a8431" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39785 } } }
I20260928 14:40:48.097257 2443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:48.097270 2447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:48.099968 2447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:48.104801 2447 catalog_manager.cc:1347] Generated new cluster ID: d28d85819c83447e9cbde67a530316ed
I20260928 14:40:48.104856 2447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:48.112251 2447 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:48.113348 2447 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:48.119652 2447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e7264e4f7cf4c9aaa45b0e6fd7a8431: Generated new TSK 0
I20260928 14:40:48.120203 2447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:40:48.148629 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39785
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:48.248685 2463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:48.248884 2463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:48.248924 2463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:48.252501 2463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:48.252547 2463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:48.252596 2463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:40:48.257390 2463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:48.258219 2463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:48.259004 2463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:48.267457 2468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:48.267477 2471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:48.267846 2463 server_base.cc:1034] running on GCE node
W20260928 14:40:48.267848 2469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:48.268229 2463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:48.268759 2463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:48.269930 2463 hybrid_clock.cc:648] HybridClock initialized: now 1790606448269916 us; error 32 us; skew 500 ppm
Sep 28 14:40:48 dist-test-slave-0n27 krb5kdc[2355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:48.272246 2463 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:40:48.273132 2463 webserver.cc:460] Webserver started at http://127.1.201.1:38133/ using document root <none> and password file <none>
I20260928 14:40:48.273627 2463 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:48.273721 2463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:48.273959 2463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:48.275744 2463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "8a0bfa90c59c4a27badf8459bfb2f720"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "a332e74f271afc2e607aab799256ec44"
server_key_iv: "77eecd63871ba150f1f5b79314107cfb"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.276248 2463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "8a0bfa90c59c4a27badf8459bfb2f720"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "a332e74f271afc2e607aab799256ec44"
server_key_iv: "77eecd63871ba150f1f5b79314107cfb"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.279523 2463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:40:48.281611 2478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:48.282622 2463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:48.282752 2463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "8a0bfa90c59c4a27badf8459bfb2f720"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "a332e74f271afc2e607aab799256ec44"
server_key_iv: "77eecd63871ba150f1f5b79314107cfb"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.282863 2463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:48.294345 2463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:48.296782 2463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:48.296931 2463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:48.297374 2463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:40:48.298205 2463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:48.298269 2463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:48.298326 2463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:48.298377 2463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:48.307044 2463 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:36555
I20260928 14:40:48.307065 2592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:36555 every 8 connection(s)
I20260928 14:40:48.307699 2463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:40:48.313551 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 2463
I20260928 14:40:48.313644 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:40:48.313928 1828 external_mini_cluster.cc:1428] Setting key 8918cd650d30d6044a508153b87cc66e
Sep 28 14:40:48 dist-test-slave-0n27 krb5kdc[2355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:48.319254 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.309083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:60381 (local address 127.1.201.62:39785)
0928 14:40:48.309196 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:48.309199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:48.309836 (+ 637us) server_negotiation.cc:407] Connection header received
0928 14:40:48.310957 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:48.310978 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:48.311021 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:48.311086 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:48.311981 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.312449 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.313337 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.313645 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.313739 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.316058 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:48.316065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:48.316071 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:48.316103 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:48.317392 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:48.317951 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:48.317954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:48.317955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:48.318020 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:48.318348 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:48.318350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:48.318351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:48.318485 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:48.318546 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:48.319046 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 14:40:48.319127 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:40:48.320133 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.309350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39785 (local address 127.1.201.1:60381)
0928 14:40:48.309700 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:48.309733 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:48.310799 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:48.311220 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:48.311225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:48.311492 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:48.311874 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.311884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.312537 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.312539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.313252 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.313260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.313768 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.313770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.313936 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.314411 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:48.314431 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:48.315883 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:48.317496 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:48.317504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:48.317518 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:48.317865 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:48.318102 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:48.318107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:48.318111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:48.318263 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:48.318572 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:48.318579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:48.318836 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 14:40:48.319010 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 14:40:48.319294 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260928 14:40:48.321221 2593 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39785
I20260928 14:40:48.321444 2593 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:48.321985 2593 heartbeater.cc:511] Master 127.1.201.62:39785 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
I20260928 14:40:48.323381 2400 ts_manager.cc:194] Registered new tserver with Master: 8a0bfa90c59c4a27badf8459bfb2f720 (127.1.201.1:36555)
I20260928 14:40:48.324345 2400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:60381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:40:48.340528 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39785
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:48.440817 2598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:48.441012 2598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:48.441049 2598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:48.444626 2598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:48.444665 2598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:48.444715 2598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:40:48.449643 2598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:48.450469 2598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:48.451280 2598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:48.459903 2604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:48.459908 2606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:48.459988 2603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:48.460130 2598 server_base.cc:1034] running on GCE node
I20260928 14:40:48.460548 2598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:48.461126 2598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:48.462273 2598 hybrid_clock.cc:648] HybridClock initialized: now 1790606448462258 us; error 28 us; skew 500 ppm
Sep 28 14:40:48 dist-test-slave-0n27 krb5kdc[2355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:48.464637 2598 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:40:48.465371 2598 webserver.cc:460] Webserver started at http://127.1.201.2:41477/ using document root <none> and password file <none>
I20260928 14:40:48.465929 2598 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:48.466019 2598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:48.466233 2598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:48.467960 2598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "ef509d99ec8d489890db46f9a243cb3a"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "1c54090bcbc2b0f2abe29e2d79efdd79"
server_key_iv: "e6e3c48982b96258c0a2552384bb1398"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.468492 2598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "ef509d99ec8d489890db46f9a243cb3a"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "1c54090bcbc2b0f2abe29e2d79efdd79"
server_key_iv: "e6e3c48982b96258c0a2552384bb1398"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.471760 2598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 14:40:48.473871 2613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:48.474932 2598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:40:48.475052 2598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "ef509d99ec8d489890db46f9a243cb3a"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "1c54090bcbc2b0f2abe29e2d79efdd79"
server_key_iv: "e6e3c48982b96258c0a2552384bb1398"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.475152 2598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:48.486300 2598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:48.488704 2598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:48.488880 2598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:48.489367 2598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:40:48.490119 2598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:48.490187 2598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:48.490250 2598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:48.490296 2598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:48.498358 2598 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:40209
I20260928 14:40:48.498379 2727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:40209 every 8 connection(s)
I20260928 14:40:48.499119 2598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:40:48.505728 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 2598
I20260928 14:40:48.505870 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:40:48.506183 1828 external_mini_cluster.cc:1428] Setting key 367e2321e1e89ad881c8b40753c5f753
Sep 28 14:40:48 dist-test-slave-0n27 krb5kdc[2355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:48.511543 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.500666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:52173 (local address 127.1.201.62:39785)
0928 14:40:48.500836 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:48.500841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:48.501530 (+ 689us) server_negotiation.cc:407] Connection header received
0928 14:40:48.502736 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:48.502739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:48.502786 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:48.502855 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:48.503795 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.504280 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.505284 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.505511 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.505585 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.508269 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:48.508272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:48.508274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:48.508294 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:48.509590 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:48.510100 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:48.510103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:48.510104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:48.510193 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:48.510506 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:48.510509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:48.510511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:48.510676 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:48.510784 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:48.511279 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 14:40:48.511390 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 14:40:48.512161 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.501006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39785 (local address 127.1.201.2:52173)
0928 14:40:48.501390 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:48.501426 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:48.502535 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:48.502990 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:48.502995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:48.503278 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:48.503660 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.503671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.504375 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.504377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.505181 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.505189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.505653 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.505658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.505891 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.506459 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:48.506474 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:48.508130 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:48.509699 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:48.509706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:48.509717 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:48.509992 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:48.510279 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:48.510284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:48.510286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:48.510408 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:48.510784 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:48.510792 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:48.511020 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 14:40:48.511223 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 14:40:48.511490 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":109,"threads_started":1}
I20260928 14:40:48.513248 2728 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39785
I20260928 14:40:48.513441 2728 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:48.513888 2728 heartbeater.cc:511] Master 127.1.201.62:39785 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
I20260928 14:40:48.514732 2400 ts_manager.cc:194] Registered new tserver with Master: ef509d99ec8d489890db46f9a243cb3a (127.1.201.2:40209)
I20260928 14:40:48.515219 2400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:52173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:40:48.532645 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39785
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:48.632259 2733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:48.632457 2733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:48.632496 2733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:48.636276 2733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:48.636322 2733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:48.636407 2733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:40:48.641093 2733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:48.641923 2733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:48.642776 2733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:48.651491 2738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:48.651562 2741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:48.651548 2733 server_base.cc:1034] running on GCE node
W20260928 14:40:48.651772 2739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:48.652139 2733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:48.652608 2733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:48.653754 2733 hybrid_clock.cc:648] HybridClock initialized: now 1790606448653739 us; error 26 us; skew 500 ppm
Sep 28 14:40:48 dist-test-slave-0n27 krb5kdc[2355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:48.656270 2733 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:40:48.657114 2733 webserver.cc:460] Webserver started at http://127.1.201.3:37185/ using document root <none> and password file <none>
I20260928 14:40:48.657603 2733 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:48.657658 2733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:48.657897 2733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:48.659611 2733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "c0e56587c3fe47fba129b578f4c781cb"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "57e851fae5bc8826b7c81f4b5d8e393f"
server_key_iv: "a6a21a795ce2c63900b2ebe31c724256"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.660091 2733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "c0e56587c3fe47fba129b578f4c781cb"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "57e851fae5bc8826b7c81f4b5d8e393f"
server_key_iv: "a6a21a795ce2c63900b2ebe31c724256"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.663146 2733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 14:40:48.665253 2748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:48.666399 2733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:40:48.666518 2733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "c0e56587c3fe47fba129b578f4c781cb"
format_stamp: "Formatted at 2026-09-28 14:40:48 on dist-test-slave-0n27"
server_key: "57e851fae5bc8826b7c81f4b5d8e393f"
server_key_iv: "a6a21a795ce2c63900b2ebe31c724256"
server_key_version: "encryptionkey@0"
I20260928 14:40:48.666616 2733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:48.681269 2733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:48.683480 2733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:48.683631 2733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:48.684067 2733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:40:48.684696 2733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:48.684736 2733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:48.684792 2733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:48.684825 2733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:48.692839 2733 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:43761
I20260928 14:40:48.692860 2861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:43761 every 8 connection(s)
I20260928 14:40:48.693583 2733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:40:48.697207 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 2733
I20260928 14:40:48.697342 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:40:48.697722 1828 external_mini_cluster.cc:1428] Setting key 7dc27bd0cf96a20c9de2356177a41315
Sep 28 14:40:48 dist-test-slave-0n27 krb5kdc[2355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:48.705715 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.695134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:54863 (local address 127.1.201.62:39785)
0928 14:40:48.695262 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:48.695265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:48.695998 (+ 733us) server_negotiation.cc:407] Connection header received
0928 14:40:48.697450 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:48.697454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:48.697545 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:48.697625 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:48.699018 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.699740 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.700744 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.700941 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.701010 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.703119 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:48.703122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:48.703124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:48.703145 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:48.704227 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:48.704607 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:48.704609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:48.704610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:48.704674 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:48.704917 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:48.704919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:48.704920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:48.705057 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:48.705122 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:48.705522 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 14:40:48.705607 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:40:48.706390 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.695471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39785 (local address 127.1.201.3:54863)
0928 14:40:48.695875 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:48.695911 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:48.697139 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:48.697825 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:48.697833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:48.698253 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:48.698843 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.698879 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.699839 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.699842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.700642 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.700650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.701049 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.701052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.701174 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.701672 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:48.701696 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:48.702989 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:48.704315 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:48.704319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:48.704328 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:48.704533 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:48.704744 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:48.704746 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:48.704747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:48.704849 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:48.705130 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:48.705134 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:48.705308 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 14:40:48.705475 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 14:40:48.705672 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
I20260928 14:40:48.707391 2862 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39785
I20260928 14:40:48.707588 2862 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:48.707984 2862 heartbeater.cc:511] Master 127.1.201.62:39785 requested a full tablet report, sending...
I20260928 14:40:48.708782 2400 ts_manager.cc:194] Registered new tserver with Master: c0e56587c3fe47fba129b578f4c781cb (127.1.201.3:43761)
I20260928 14:40:48.709226 2400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:54863
I20260928 14:40:48.711192 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:40:48.720242 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.714033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.1.201.62:39785)
0928 14:40:48.714147 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:48.714151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:48.714321 (+ 170us) server_negotiation.cc:407] Connection header received
0928 14:40:48.714389 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:48.714391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:48.714421 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:48.714471 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:48.715021 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.715486 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.716232 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.716420 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.716488 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.717644 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:48.717647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:48.717649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:48.717677 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:48.718804 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:48.719186 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:48.719190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:48.719194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:48.719268 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:48.719512 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:48.719515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:48.719517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:48.719701 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:48.719825 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:48.720031 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 14:40:48.720121 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:40:48.723941 2400 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:40:48.726696 2400 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:40:48.738931 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.733803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:43761 (local address 127.0.0.1:40698)
0928 14:40:48.734442 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:48.734453 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:48.734520 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:48.735209 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:48.735212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:48.735229 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:48.735325 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.735332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.736358 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.736363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.737844 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.737851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.738567 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.738570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.738653 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.738678 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:40:48.738768 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 14:40:48.738842 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":169,"thread_start_us":79,"threads_started":1}
I20260928 14:40:48.739032 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.734229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.1.201.3:43761)
0928 14:40:48.734804 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:48.734819 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:48.734836 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:40:48.734892 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:48.734896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:48.735058 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:48.735220 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:48.735483 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.736238 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.737960 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.738463 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.738545 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.738630 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:48.738820 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 14:40:48.738902 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":79,"threads_started":1}
I20260928 14:40:48.739871 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.733616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:36555 (local address 127.0.0.1:58096)
0928 14:40:48.734072 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:48.734113 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:48.734219 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:48.734915 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:48.734920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:48.734939 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:48.735047 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.735052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.736445 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.736447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.737841 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.737851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.739512 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.739515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.739601 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.739621 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:40:48.739697 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:40:48.739761 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":77,"thread_start_us":65,"threads_started":1}
I20260928 14:40:48.739943 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.733758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.1.201.1:36555)
0928 14:40:48.734164 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:48.734178 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:48.734200 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:40:48.734407 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:48.734414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:48.734599 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:48.734737 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:48.735160 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.736338 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.738687 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.739401 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.739478 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.739558 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:48.739738 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 14:40:48.739814 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":114,"threads_started":1}
I20260928 14:40:48.740818 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.734195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:40209 (local address 127.0.0.1:42928)
0928 14:40:48.734673 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:48.734698 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:48.734763 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:48.735439 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:48.735442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:48.735462 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:48.735535 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.735540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.736789 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.736795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.737842 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.737852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.740466 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.740469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.740553 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.740575 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:40:48.740620 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 14:40:48.740688 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":48,"thread_start_us":75,"threads_started":1}
I20260928 14:40:48.741410 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.734419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.1.201.2:40209)
0928 14:40:48.734977 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:48.734987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:48.735006 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:40:48.735072 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:48.735079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:48.735249 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:48.735443 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:48.735646 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.736657 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.739601 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.740302 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.741049 (+ 747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.741165 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:48.741208 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 14:40:48.741278 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":115,"threads_started":1}
I20260928 14:40:48.741698 2796 tablet_service.cc:1467] Processing CreateTablet for tablet ae53e40b67764b65b16f8404951d2b1b (DEFAULT_TABLE table=test-table [id=80159cbb06e943afafb78ea0c153043c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:48.742640 2796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae53e40b67764b65b16f8404951d2b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:48.742552 2526 tablet_service.cc:1467] Processing CreateTablet for tablet ae53e40b67764b65b16f8404951d2b1b (DEFAULT_TABLE table=test-table [id=80159cbb06e943afafb78ea0c153043c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:48.743427 2526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae53e40b67764b65b16f8404951d2b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:48.744122 2661 tablet_service.cc:1467] Processing CreateTablet for tablet ae53e40b67764b65b16f8404951d2b1b (DEFAULT_TABLE table=test-table [id=80159cbb06e943afafb78ea0c153043c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:48.745049 2661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae53e40b67764b65b16f8404951d2b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:48.749377 2880 tablet_bootstrap.cc:492] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720: Bootstrap starting.
I20260928 14:40:48.750186 2881 tablet_bootstrap.cc:492] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb: Bootstrap starting.
I20260928 14:40:48.751441 2882 tablet_bootstrap.cc:492] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a: Bootstrap starting.
I20260928 14:40:48.752027 2881 tablet_bootstrap.cc:654] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:48.752117 2880 tablet_bootstrap.cc:654] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:48.752907 2880 log.cc:824] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:48.752954 2881 log.cc:824] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:48.753965 2882 tablet_bootstrap.cc:654] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:48.754496 2882 log.cc:824] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:48.754647 2881 tablet_bootstrap.cc:492] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb: No bootstrap required, opened a new log
I20260928 14:40:48.754647 2880 tablet_bootstrap.cc:492] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720: No bootstrap required, opened a new log
I20260928 14:40:48.754813 2881 ts_tablet_manager.cc:1397] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:40:48.754813 2880 ts_tablet_manager.cc:1397] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:40:48.756265 2882 tablet_bootstrap.cc:492] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a: No bootstrap required, opened a new log
I20260928 14:40:48.756456 2882 ts_tablet_manager.cc:1397] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:40:48.758476 2881 raft_consensus.cc:348] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36555 } } peers { permanent_uuid: "c0e56587c3fe47fba129b578f4c781cb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43761 } } peers { permanent_uuid: "ef509d99ec8d489890db46f9a243cb3a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40209 } }
I20260928 14:40:48.758476 2880 raft_consensus.cc:348] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36555 } } peers { permanent_uuid: "c0e56587c3fe47fba129b578f4c781cb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43761 } } peers { permanent_uuid: "ef509d99ec8d489890db46f9a243cb3a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40209 } }
I20260928 14:40:48.758718 2881 raft_consensus.cc:374] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:48.758718 2880 raft_consensus.cc:374] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:48.758771 2881 raft_consensus.cc:729] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e56587c3fe47fba129b578f4c781cb, State: Initialized, Role: FOLLOWER
I20260928 14:40:48.758771 2880 raft_consensus.cc:729] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a0bfa90c59c4a27badf8459bfb2f720, State: Initialized, Role: FOLLOWER
I20260928 14:40:48.759166 2880 consensus_queue.cc:260] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36555 } } peers { permanent_uuid: "c0e56587c3fe47fba129b578f4c781cb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43761 } } peers { permanent_uuid: "ef509d99ec8d489890db46f9a243cb3a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40209 } }
I20260928 14:40:48.759166 2881 consensus_queue.cc:260] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36555 } } peers { permanent_uuid: "c0e56587c3fe47fba129b578f4c781cb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43761 } } peers { permanent_uuid: "ef509d99ec8d489890db46f9a243cb3a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40209 } }
I20260928 14:40:48.759550 2882 raft_consensus.cc:348] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36555 } } peers { permanent_uuid: "c0e56587c3fe47fba129b578f4c781cb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43761 } } peers { permanent_uuid: "ef509d99ec8d489890db46f9a243cb3a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40209 } }
I20260928 14:40:48.759783 2882 raft_consensus.cc:374] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:48.759835 2882 raft_consensus.cc:729] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef509d99ec8d489890db46f9a243cb3a, State: Initialized, Role: FOLLOWER
I20260928 14:40:48.760054 2881 ts_tablet_manager.cc:1428] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:40:48.760054 2880 ts_tablet_manager.cc:1428] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260928 14:40:48.760087 2862 heartbeater.cc:503] Master 127.1.201.62:39785 was elected leader, sending a full tablet report...
I20260928 14:40:48.760252 2882 consensus_queue.cc:260] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36555 } } peers { permanent_uuid: "c0e56587c3fe47fba129b578f4c781cb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43761 } } peers { permanent_uuid: "ef509d99ec8d489890db46f9a243cb3a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40209 } }
I20260928 14:40:48.760762 2728 heartbeater.cc:503] Master 127.1.201.62:39785 was elected leader, sending a full tablet report...
I20260928 14:40:48.761067 2882 ts_tablet_manager.cc:1428] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:40:48.761067 2593 heartbeater.cc:503] Master 127.1.201.62:39785 was elected leader, sending a full tablet report...
W20260928 14:40:48.808637 2594 tablet.cc:2369] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:40:48.944495 2863 tablet.cc:2369] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:40:48.948130 2886 raft_consensus.cc:482] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:40:48.948343 2886 raft_consensus.cc:504] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36555 } } peers { permanent_uuid: "c0e56587c3fe47fba129b578f4c781cb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43761 } } peers { permanent_uuid: "ef509d99ec8d489890db46f9a243cb3a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40209 } }
I20260928 14:40:48.949575 2886 leader_election.cc:288] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0e56587c3fe47fba129b578f4c781cb (127.1.201.3:43761), ef509d99ec8d489890db46f9a243cb3a (127.1.201.2:40209)
I20260928 14:40:48.953189 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.949986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:38793 (local address 127.1.201.2:40209)
0928 14:40:48.950108 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:48.950112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:48.950200 (+ 88us) server_negotiation.cc:407] Connection header received
0928 14:40:48.950355 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:48.950359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:48.950411 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:48.950495 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:48.950790 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.951296 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.952291 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.952672 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.952785 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.952843 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:48.953009 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 14:40:48.953076 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:40:48.953203 2889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.949843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:40209 (local address 127.1.201.1:38793)
0928 14:40:48.950098 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:48.950111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:48.950238 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:48.950509 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:48.950512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:48.950564 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:48.950683 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.950688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.951408 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.951412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.952175 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.952184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.952787 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.952790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.952881 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.952910 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:40:48.952994 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:40:48.953058 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"spinlock_wait_cycles":49408,"thread_start_us":82,"threads_started":1}
I20260928 14:40:48.953550 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.949911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:42061 (local address 127.1.201.3:43761)
0928 14:40:48.950071 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:48.950076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:48.950502 (+ 426us) server_negotiation.cc:407] Connection header received
0928 14:40:48.950677 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:48.950680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:48.950730 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:48.950812 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:48.951075 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.951565 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.952616 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.952994 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.953107 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.953161 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:48.953382 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 14:40:48.953446 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:40:48.953855 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae53e40b67764b65b16f8404951d2b1b" candidate_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef509d99ec8d489890db46f9a243cb3a" is_pre_election: true
I20260928 14:40:48.953945 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:48.949844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:43761 (local address 127.1.201.1:42061)
0928 14:40:48.950408 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:48.950420 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:48.950571 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:48.950824 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:48.950826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:48.950842 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:48.950963 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.950968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.951676 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.951679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.952498 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:48.952507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:48.953110 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:48.953114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:48.953206 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:48.953234 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 14:40:48.953718 (+ 484us) client_negotiation.cc:240] Negotiation successful
0928 14:40:48.953824 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":283,"spinlock_wait_cycles":5760,"thread_start_us":101,"threads_started":1}
I20260928 14:40:48.954144 2681 raft_consensus.cc:2454] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a0bfa90c59c4a27badf8459bfb2f720 in term 0.
I20260928 14:40:48.954487 2816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae53e40b67764b65b16f8404951d2b1b" candidate_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e56587c3fe47fba129b578f4c781cb" is_pre_election: true
I20260928 14:40:48.954629 2479 leader_election.cc:302] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a0bfa90c59c4a27badf8459bfb2f720, ef509d99ec8d489890db46f9a243cb3a; no voters:
I20260928 14:40:48.954721 2816 raft_consensus.cc:2454] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a0bfa90c59c4a27badf8459bfb2f720 in term 0.
I20260928 14:40:48.954902 2886 raft_consensus.cc:2781] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:40:48.955008 2886 raft_consensus.cc:482] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:40:48.955066 2886 raft_consensus.cc:3037] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:48.956110 2886 raft_consensus.cc:504] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36555 } } peers { permanent_uuid: "c0e56587c3fe47fba129b578f4c781cb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43761 } } peers { permanent_uuid: "ef509d99ec8d489890db46f9a243cb3a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40209 } }
I20260928 14:40:48.956426 2886 leader_election.cc:288] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [CANDIDATE]: Term 1 election: Requested vote from peers c0e56587c3fe47fba129b578f4c781cb (127.1.201.3:43761), ef509d99ec8d489890db46f9a243cb3a (127.1.201.2:40209)
I20260928 14:40:48.956687 2816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae53e40b67764b65b16f8404951d2b1b" candidate_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e56587c3fe47fba129b578f4c781cb"
I20260928 14:40:48.956795 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae53e40b67764b65b16f8404951d2b1b" candidate_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef509d99ec8d489890db46f9a243cb3a"
I20260928 14:40:48.956826 2816 raft_consensus.cc:3037] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:48.956928 2681 raft_consensus.cc:3037] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:48.957947 2816 raft_consensus.cc:2454] T ae53e40b67764b65b16f8404951d2b1b P c0e56587c3fe47fba129b578f4c781cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a0bfa90c59c4a27badf8459bfb2f720 in term 1.
I20260928 14:40:48.958141 2681 raft_consensus.cc:2454] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a0bfa90c59c4a27badf8459bfb2f720 in term 1.
I20260928 14:40:48.958449 2479 leader_election.cc:302] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a0bfa90c59c4a27badf8459bfb2f720, ef509d99ec8d489890db46f9a243cb3a; no voters:
I20260928 14:40:48.958647 2886 raft_consensus.cc:2781] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:48.958881 2886 raft_consensus.cc:686] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [term 1 LEADER]: Becoming Leader. State: Replica: 8a0bfa90c59c4a27badf8459bfb2f720, State: Running, Role: LEADER
I20260928 14:40:48.959187 2886 consensus_queue.cc:237] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36555 } } peers { permanent_uuid: "c0e56587c3fe47fba129b578f4c781cb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43761 } } peers { permanent_uuid: "ef509d99ec8d489890db46f9a243cb3a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40209 } }
I20260928 14:40:48.962172 2400 catalog_manager.cc:5495] T ae53e40b67764b65b16f8404951d2b1b P 8a0bfa90c59c4a27badf8459bfb2f720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a0bfa90c59c4a27badf8459bfb2f720 (127.1.201.1). New cstate: current_term: 1 leader_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0bfa90c59c4a27badf8459bfb2f720" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0e56587c3fe47fba129b578f4c781cb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef509d99ec8d489890db46f9a243cb3a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40209 } health_report { overall_health: UNKNOWN } } }
W20260928 14:40:49.000097 2729 tablet.cc:2369] T ae53e40b67764b65b16f8404951d2b1b P ef509d99ec8d489890db46f9a243cb3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 14:40:49 dist-test-slave-0n27 krb5kdc[2355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:40:49.028625 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:49.020078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.1.201.1:36555)
0928 14:40:49.020253 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:49.020259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:49.020403 (+ 144us) server_negotiation.cc:407] Connection header received
0928 14:40:49.020555 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:49.020558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:49.020598 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:49.020688 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:49.021367 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:49.021957 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:49.022805 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:49.023020 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:49.023126 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:49.025419 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:49.025435 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:49.025448 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:49.025469 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:49.026878 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:49.027329 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:49.027334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:49.027335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:49.027418 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:49.027713 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:49.027716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:49.027717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:49.027973 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:49.028196 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:49.028351 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 14:40:49.028472 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 14:40:49 dist-test-slave-0n27 krb5kdc[2355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:40:49 dist-test-slave-0n27 krb5kdc[2355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:40:49.052989 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:49.046117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.1.201.1:36555)
0928 14:40:49.046275 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:49.046279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:49.046383 (+ 104us) server_negotiation.cc:407] Connection header received
0928 14:40:49.046472 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:49.046474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:49.046521 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:49.046584 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:49.047135 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:49.047646 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:49.048356 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:49.048540 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:49.048610 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:49.050621 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:49.050624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:49.050626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:49.050644 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:49.051682 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:49.052017 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:49.052019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:49.052020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:49.052083 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:49.052298 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:49.052300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:49.052301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:49.052426 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:49.052534 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:49.052712 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 14:40:49.052799 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 14:40:49 dist-test-slave-0n27 krb5kdc[2355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:40:49 dist-test-slave-0n27 krb5kdc[2355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:40:49.075181 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:49.067870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.1.201.1:36555)
0928 14:40:49.067972 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:49.067978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:49.068112 (+ 134us) server_negotiation.cc:407] Connection header received
0928 14:40:49.068251 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:49.068253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:49.068284 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:49.068355 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:49.069017 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:49.069542 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:49.070355 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:49.070627 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:49.070727 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:49.072688 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:49.072691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:49.072694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:49.072715 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:49.073713 (+ 998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:49.074100 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:49.074104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:49.074106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:49.074179 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:49.074472 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:49.074475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:49.074477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:49.074615 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:49.074749 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:49.074952 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 14:40:49.075047 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 14:40:49.076642 2506 tablet_service.cc:2908] Rejecting scan request for tablet ae53e40b67764b65b16f8404951d2b1b: Uninitialized: clean time has not yet been initialized
I20260928 14:40:49.078828 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 2463
I20260928 14:40:49.084806 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 2598
I20260928 14:40:49.091012 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 2733
I20260928 14:40:49.096226 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 2368
2026-09-28T14:40:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2225 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:49 dist-test-slave-0n27 krb5kdc[2905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:49 dist-test-slave-0n27 krb5kdc[2905](info): set up 2 sockets
Sep 28 14:40:49 dist-test-slave-0n27 krb5kdc[2905](info): commencing operation
krb5kdc: starting...
W20260928 14:40:50.058434 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:40:50 dist-test-slave-0n27 krb5kdc[2905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:40:50.168112 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:44903
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:44903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:50.269294 2918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:50.269500 2918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:50.269531 2918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:50.273072 2918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:50.273113 2918 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:40:50.273126 2918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:50.273161 2918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:50.273203 2918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:50.273233 2918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:50.278067 2918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:44903
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:44903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:50.278856 2918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:50.279673 2918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:50.287045 2923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:50.287058 2924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:50.287104 2918 server_base.cc:1034] running on GCE node
W20260928 14:40:50.287058 2926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:50.287722 2918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:50.288568 2918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:50.289713 2918 hybrid_clock.cc:648] HybridClock initialized: now 1790606450289691 us; error 33 us; skew 500 ppm
Sep 28 14:40:50 dist-test-slave-0n27 krb5kdc[2905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606450, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:50.299592 2918 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:40:50.300416 2918 webserver.cc:460] Webserver started at http://127.1.201.62:43221/ using document root <none> and password file <none>
I20260928 14:40:50.300902 2918 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:50.300959 2918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:50.301187 2918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:50.303032 2918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "4f35ca77aa7a48d491b3f77d29ba5161"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "7c49f6f768e2a9d5e450b36beaa0b93d"
server_key_iv: "30d5af37c707a4e845d6ede048a73dd7"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.303521 2918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "4f35ca77aa7a48d491b3f77d29ba5161"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "7c49f6f768e2a9d5e450b36beaa0b93d"
server_key_iv: "30d5af37c707a4e845d6ede048a73dd7"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.306726 2918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:40:50.309119 2933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:50.310469 2918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:40:50.310585 2918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "4f35ca77aa7a48d491b3f77d29ba5161"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "7c49f6f768e2a9d5e450b36beaa0b93d"
server_key_iv: "30d5af37c707a4e845d6ede048a73dd7"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.310681 2918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:50.329067 2918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:50.331591 2918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:50.331776 2918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:50.338785 2918 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:44903
I20260928 14:40:50.338809 2996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:44903 every 8 connection(s)
I20260928 14:40:50.339546 2918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:50.341953 2997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:50.342898 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 2918
I20260928 14:40:50.343016 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:40:50.343303 1828 external_mini_cluster.cc:1428] Setting key 5663dcdd42c883ffce7a9941c08a9317
I20260928 14:40:50.347496 2997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161: Bootstrap starting.
I20260928 14:40:50.349448 2997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:50.350304 2997 log.cc:824] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:40:50 dist-test-slave-0n27 krb5kdc[2905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:50.352233 2997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161: No bootstrap required, opened a new log
I20260928 14:40:50.355743 2997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f35ca77aa7a48d491b3f77d29ba5161" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44903 } }
I20260928 14:40:50.355929 2997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:50.355952 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.344641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.1.201.62:44903)
0928 14:40:50.345039 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:50.345053 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:50.345083 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:40:50.345675 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:50.345689 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:50.346048 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:50.346353 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:50.347129 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.348080 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.348867 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.349300 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.349477 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:50.352048 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:50.352052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:50.352063 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:50.352088 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:50.353485 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:50.353945 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:50.353949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:50.353952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:50.354046 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:50.354308 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:50.354311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:50.354312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:50.354607 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:50.354760 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:50.354994 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 14:40:50.355242 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260928 14:40:50.355962 2997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f35ca77aa7a48d491b3f77d29ba5161, State: Initialized, Role: FOLLOWER
I20260928 14:40:50.356423 2997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f35ca77aa7a48d491b3f77d29ba5161" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44903 } }
I20260928 14:40:50.356544 2997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:50.356624 2997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:50.356753 2997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:50.357622 2997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f35ca77aa7a48d491b3f77d29ba5161" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44903 } }
I20260928 14:40:50.357935 2997 leader_election.cc:302] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f35ca77aa7a48d491b3f77d29ba5161; no voters:
I20260928 14:40:50.358197 2997 leader_election.cc:288] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:50.358260 3002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:50.358490 3002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [term 1 LEADER]: Becoming Leader. State: Replica: 4f35ca77aa7a48d491b3f77d29ba5161, State: Running, Role: LEADER
I20260928 14:40:50.358793 3002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f35ca77aa7a48d491b3f77d29ba5161" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44903 } }
I20260928 14:40:50.359274 2997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:50.360452 3003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f35ca77aa7a48d491b3f77d29ba5161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f35ca77aa7a48d491b3f77d29ba5161" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44903 } } }
I20260928 14:40:50.360477 3004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f35ca77aa7a48d491b3f77d29ba5161. Latest consensus state: current_term: 1 leader_uuid: "4f35ca77aa7a48d491b3f77d29ba5161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f35ca77aa7a48d491b3f77d29ba5161" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44903 } } }
I20260928 14:40:50.360584 3003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:50.360592 3004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:50.361246 3012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:50.363970 3012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:50.368593 3012 catalog_manager.cc:1347] Generated new cluster ID: df886ff8a56e44acb36e3f2a2c8d821f
I20260928 14:40:50.368643 3012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:50.396466 3012 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:50.397608 3012 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:50.413877 3012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161: Generated new TSK 0
I20260928 14:40:50.414547 3012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:40:50.439921 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:44903
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:50.539065 3024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:50.539268 3024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:50.539307 3024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:50.543092 3024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:50.543136 3024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:50.543208 3024 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:40:50.543262 3024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:40:50.548113 3024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.201.62:44903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:50.548949 3024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:50.549823 3024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:50.558617 3029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:50.558636 3032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:50.558636 3030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:50.558879 3024 server_base.cc:1034] running on GCE node
I20260928 14:40:50.559514 3024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:50.560088 3024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:50.561286 3024 hybrid_clock.cc:648] HybridClock initialized: now 1790606450561268 us; error 32 us; skew 500 ppm
Sep 28 14:40:50 dist-test-slave-0n27 krb5kdc[2905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606450, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:50.563689 3024 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:40:50.564455 3024 webserver.cc:460] Webserver started at http://127.1.201.1:40327/ using document root <none> and password file <none>
I20260928 14:40:50.564949 3024 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:50.565034 3024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:50.565254 3024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:50.567188 3024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "0671218a17394bfb9285762f316b5606"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "5b1f682c9696d6ee00cf3553ad32987b"
server_key_iv: "a21f2031890a0281068f376241be6dab"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.567724 3024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "0671218a17394bfb9285762f316b5606"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "5b1f682c9696d6ee00cf3553ad32987b"
server_key_iv: "a21f2031890a0281068f376241be6dab"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.570978 3024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:40:50.573150 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:50.574196 3024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:50.574323 3024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "0671218a17394bfb9285762f316b5606"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "5b1f682c9696d6ee00cf3553ad32987b"
server_key_iv: "a21f2031890a0281068f376241be6dab"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.574421 3024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:50.582835 3024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:50.585289 3024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:50.585467 3024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:50.586794 3024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:50.586863 3024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:50.586932 3024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:50.586977 3024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:40:50 dist-test-slave-0n27 krb5kdc[2905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606450, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:50.599664 3024 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:33483
I20260928 14:40:50.599834 3156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:33483 every 8 connection(s)
I20260928 14:40:50.600590 3024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:40:50.601032 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.587540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:47269 (local address 127.1.201.62:44903)
0928 14:40:50.587654 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:50.587658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:50.588556 (+ 898us) server_negotiation.cc:407] Connection header received
0928 14:40:50.589511 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:50.589515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:50.589581 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:50.589640 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:50.590993 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.591605 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.593035 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.593333 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.593443 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:50.597238 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:50.597243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:50.597246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:50.597273 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:50.598809 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:50.599408 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:50.599414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:50.599419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:50.599510 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:50.599897 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:50.599900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:50.599901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:50.600092 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:50.600180 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:50.600755 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 14:40:50.600859 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:40:50.601819 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.587964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44903 (local address 127.1.201.1:47269)
0928 14:40:50.588422 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:50.588458 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:50.589268 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:50.590063 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:50.590069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:50.590347 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:50.590833 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:50.590845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.591733 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.591736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:50.592907 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:50.592917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.593473 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.593477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:50.593642 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:50.594192 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:50.594217 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:50.597049 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:50.598934 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:50.598938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:50.598949 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:50.599273 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:50.599637 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:50.599640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:50.599642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:50.599790 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:50.600228 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:50.600233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:50.600484 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 14:40:50.600688 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 14:40:50.600990 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":140,"threads_started":1}
I20260928 14:40:50.603134 3157 heartbeater.cc:347] Connected to a master server at 127.1.201.62:44903
I20260928 14:40:50.603283 3157 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:50.603724 3157 heartbeater.cc:511] Master 127.1.201.62:44903 requested a full tablet report, sending...
I20260928 14:40:50.604833 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 3024
I20260928 14:40:50.604964 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:40:50.605222 2950 ts_manager.cc:194] Registered new tserver with Master: 0671218a17394bfb9285762f316b5606 (127.1.201.1:33483)
I20260928 14:40:50.605253 1828 external_mini_cluster.cc:1428] Setting key 71354206bcbcfcc42ae51f798718b251
I20260928 14:40:50.606618 2950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:47269
I20260928 14:40:50.612735 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.605114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:45075 (local address 127.1.201.62:44903)
0928 14:40:50.605223 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:50.605227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:50.605827 (+ 600us) server_negotiation.cc:407] Connection header received
0928 14:40:50.605996 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:50.605999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:50.606041 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:50.606122 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:50.606635 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.607268 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.608351 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.608659 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.608730 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:50.609961 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:50.609964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:50.609965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:50.609985 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:50.611279 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:50.611751 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:50.611753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:50.611754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:50.611815 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:50.612087 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:50.612089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:50.612090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:50.612218 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:50.612282 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:50.612567 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 14:40:50.612640 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:40:50.612849 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.605302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44903 (local address 127.1.201.1:45075)
0928 14:40:50.605719 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:50.605730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:50.605886 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:50.606151 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:50.606154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:50.606349 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:50.606472 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:50.606478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.607389 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.607392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:50.608218 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:50.608227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.608865 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.608867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:50.608928 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:50.609357 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:50.609373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:50.609842 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:50.611388 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:50.611394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:50.611397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:50.611654 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:50.611896 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:50.611900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:50.611904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:50.611999 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:50.612305 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:50.612310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:50.612410 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 14:40:50.612521 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:40:50.612651 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:40:50.632493 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:44903
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:50.737637 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:50.737886 3166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:50.737932 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:50.741763 3166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:50.741820 3166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:50.741855 3166 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:40:50.741885 3166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:40:50.746765 3166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.201.62:44903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:50.747632 3166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:50.748478 3166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:50.757578 3171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:50.757566 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:50.757566 3172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:50.757939 3166 server_base.cc:1034] running on GCE node
I20260928 14:40:50.758332 3166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:50.758873 3166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:50.760061 3166 hybrid_clock.cc:648] HybridClock initialized: now 1790606450760041 us; error 28 us; skew 500 ppm
Sep 28 14:40:50 dist-test-slave-0n27 krb5kdc[2905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606450, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:50.762419 3166 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:40:50.763170 3166 webserver.cc:460] Webserver started at http://127.1.201.2:44785/ using document root <none> and password file <none>
I20260928 14:40:50.763674 3166 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:50.763767 3166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:50.763980 3166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:50.765914 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "14c79ccee5f24f9a9cffc9f18de6c55a"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "e57a74f16e021f6a9f3b207e0ff96499"
server_key_iv: "c4615433caf1c6b8d8c2030214629ced"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.766443 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "14c79ccee5f24f9a9cffc9f18de6c55a"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "e57a74f16e021f6a9f3b207e0ff96499"
server_key_iv: "c4615433caf1c6b8d8c2030214629ced"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.769886 3166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:40:50.772006 3181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:50.773260 3166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:50.773375 3166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "14c79ccee5f24f9a9cffc9f18de6c55a"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "e57a74f16e021f6a9f3b207e0ff96499"
server_key_iv: "c4615433caf1c6b8d8c2030214629ced"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.773488 3166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:50.789352 3166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:50.791791 3166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:50.791917 3166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:50.793115 3166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:50.793162 3166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:50.793196 3166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:50.793211 3166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:40:50 dist-test-slave-0n27 krb5kdc[2905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606450, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:50.803931 3166 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:46145
I20260928 14:40:50.803937 3298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:46145 every 8 connection(s)
I20260928 14:40:50.804859 3166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:40:50.806097 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.793776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:49337 (local address 127.1.201.62:44903)
0928 14:40:50.793923 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:50.793927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:50.795373 (+ 1446us) server_negotiation.cc:407] Connection header received
0928 14:40:50.796389 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:50.796395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:50.796464 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:50.796570 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:50.797806 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.798373 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.799405 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.799655 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.799785 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:50.802248 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:50.802251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:50.802254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:50.802282 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:50.803940 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:50.804473 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:50.804476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:50.804478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:50.804558 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:50.804883 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:50.804885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:50.804886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:50.805072 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:50.805181 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:50.805749 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 14:40:50.805920 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:40:50.806730 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.794738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44903 (local address 127.1.201.2:49337)
0928 14:40:50.795229 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:50.795276 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:50.796179 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:50.796765 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:50.796771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:50.797106 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:50.797645 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:50.797659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.798491 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.798494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:50.799281 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:50.799289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.799825 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.799829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:50.799983 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:50.800515 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:50.800540 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:50.802099 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:50.804060 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:50.804065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:50.804076 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:50.804361 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:50.804650 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:50.804652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:50.804654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:50.804798 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:50.805172 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:50.805177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:50.805444 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 14:40:50.805661 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 14:40:50.805981 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":123,"threads_started":1}
I20260928 14:40:50.807745 3299 heartbeater.cc:347] Connected to a master server at 127.1.201.62:44903
I20260928 14:40:50.807873 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 3166
I20260928 14:40:50.807899 3299 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:50.807987 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:40:50.808285 1828 external_mini_cluster.cc:1428] Setting key cf505edb44283540b5110a5425d34eb3
I20260928 14:40:50.808375 3299 heartbeater.cc:511] Master 127.1.201.62:44903 requested a full tablet report, sending...
I20260928 14:40:50.809381 2950 ts_manager.cc:194] Registered new tserver with Master: 14c79ccee5f24f9a9cffc9f18de6c55a (127.1.201.2:46145)
I20260928 14:40:50.810144 2950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:49337
I20260928 14:40:50.817895 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.809959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:58515 (local address 127.1.201.62:44903)
0928 14:40:50.810246 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:50.810251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:50.810386 (+ 135us) server_negotiation.cc:407] Connection header received
0928 14:40:50.810551 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:50.810557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:50.810608 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:50.810689 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:50.811366 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.812050 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.813256 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.813646 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.813767 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:50.815009 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:50.815011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:50.815013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:50.815031 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:50.816339 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:50.816819 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:50.816821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:50.816822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:50.816885 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:50.817132 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:50.817133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:50.817134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:50.817302 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:50.817370 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:50.817649 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 14:40:50.817724 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260928 14:40:50.817947 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.809976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44903 (local address 127.1.201.2:58515)
0928 14:40:50.810303 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:50.810325 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:50.810443 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:50.810809 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:50.810815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:50.811035 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:50.811237 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:50.811247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.812194 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.812200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:50.813124 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:50.813137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:50.813818 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:50.813825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:50.814000 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:50.814437 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:50.814455 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:50.814899 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:50.816446 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:50.816452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:50.816456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:50.816699 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:50.816961 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:50.816966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:50.816967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:50.817052 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:50.817393 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:50.817398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:50.817503 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 14:40:50.817610 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:40:50.817733 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":4,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:40:50.836176 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:44903
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:50.936388 3308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:50.936622 3308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:50.936697 3308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:50.940377 3308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:50.940448 3308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:50.940510 3308 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:40:50.940580 3308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:40:50.945120 3308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.201.62:44903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:50.945971 3308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:50.946842 3308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:50.955169 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:50.955183 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:50.955385 3308 server_base.cc:1034] running on GCE node
W20260928 14:40:50.955477 3316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:50.955838 3308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:50.956387 3308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:50.957562 3308 hybrid_clock.cc:648] HybridClock initialized: now 1790606450957545 us; error 29 us; skew 500 ppm
Sep 28 14:40:50 dist-test-slave-0n27 krb5kdc[2905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606450, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:50.959995 3308 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:40:50.960819 3308 webserver.cc:460] Webserver started at http://127.1.201.3:37503/ using document root <none> and password file <none>
I20260928 14:40:50.961279 3308 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:50.961359 3308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:50.961560 3308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:50.963353 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "5bc694b2a34d44f4b63b40790f3741cd"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "aa493b342c6c7c10d9749e8021d7205e"
server_key_iv: "3dd7a189fd33b29049e093206e7d1e01"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.963883 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "5bc694b2a34d44f4b63b40790f3741cd"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "aa493b342c6c7c10d9749e8021d7205e"
server_key_iv: "3dd7a189fd33b29049e093206e7d1e01"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.967602 3308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:40:50.969704 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:50.970767 3308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:50.970918 3308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "5bc694b2a34d44f4b63b40790f3741cd"
format_stamp: "Formatted at 2026-09-28 14:40:50 on dist-test-slave-0n27"
server_key: "aa493b342c6c7c10d9749e8021d7205e"
server_key_iv: "3dd7a189fd33b29049e093206e7d1e01"
server_key_version: "encryptionkey@0"
I20260928 14:40:50.971017 3308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:50.991153 3308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:50.993803 3308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:50.993986 3308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:50.995270 3308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:50.995349 3308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:50.995421 3308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:50.995466 3308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:40:51 dist-test-slave-0n27 krb5kdc[2905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606450, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:51.006590 3308 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:36033
I20260928 14:40:51.006614 3439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:36033 every 8 connection(s)
I20260928 14:40:51.007586 3308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:40:51.009284 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.996084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:43577 (local address 127.1.201.62:44903)
0928 14:40:50.996272 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:50.996280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:50.997424 (+ 1144us) server_negotiation.cc:407] Connection header received
0928 14:40:50.998522 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:50.998525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:50.998587 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:50.998692 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:50.999935 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.000478 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.001694 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.002036 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.002162 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.005312 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:51.005316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:51.005323 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:51.005354 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:51.007015 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.007598 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.007607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.007609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.007733 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.008126 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.008129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.008131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.008388 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:51.008510 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.008985 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.009110 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:40:51.009997 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:50.996807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44903 (local address 127.1.201.3:43577)
0928 14:40:50.997235 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:50.997290 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:50.998317 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:50.998864 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:50.998870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:50.999175 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:50.999787 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:50.999803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.000604 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.000608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.001556 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.001570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.002195 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.002199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.002374 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.003143 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:51.003171 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:51.005105 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:51.007142 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:51.007148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:51.007162 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:51.007487 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:51.007807 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:51.007810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:51.007812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:51.007985 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:51.008519 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:51.008528 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:51.008760 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.008940 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.009218 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":128,"threads_started":1}
I20260928 14:40:51.011179 3440 heartbeater.cc:347] Connected to a master server at 127.1.201.62:44903
I20260928 14:40:51.011220 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 3308
I20260928 14:40:51.011337 3440 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:51.011348 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:40:51.011605 1828 external_mini_cluster.cc:1428] Setting key 8063111e0646563af35eb4aa0bfd0a74
I20260928 14:40:51.011874 3440 heartbeater.cc:511] Master 127.1.201.62:44903 requested a full tablet report, sending...
I20260928 14:40:51.012940 2951 ts_manager.cc:194] Registered new tserver with Master: 5bc694b2a34d44f4b63b40790f3741cd (127.1.201.3:36033)
I20260928 14:40:51.013629 2951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:43577
I20260928 14:40:51.014633 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:40:51.021879 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.014442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:47119 (local address 127.1.201.62:44903)
0928 14:40:51.014558 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.014561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.014573 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:40:51.014607 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.014612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.014651 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.014747 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:51.015346 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.016090 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.016988 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.017373 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.017490 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.018674 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:51.018677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:51.018680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:51.018700 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:51.020305 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.020706 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.020710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.020712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.020792 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.021076 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.021079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.021081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.021259 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:51.021359 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.021608 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.021710 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:40:51.021878 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.013291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44903 (local address 127.1.201.3:47119)
0928 14:40:51.013545 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.013566 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.013631 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.014857 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.014860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.015054 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:51.015212 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.015219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.016246 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.016249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.016865 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.016874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.017511 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.017517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.017640 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.018101 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:51.018119 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:51.018542 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:51.020418 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:51.020421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:51.020424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:51.020607 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:51.020884 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:51.020888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:51.020890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:51.020979 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:51.021360 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:51.021363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:51.021452 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.021557 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.021683 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
Sep 28 14:40:51 dist-test-slave-0n27 krb5kdc[2905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:40:51 dist-test-slave-0n27 krb5kdc[2905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:51.040423 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.032689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.1.201.62:44903)
0928 14:40:51.032832 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.032836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.032845 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:40:51.032873 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.032875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.032906 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.032989 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:51.033633 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.034180 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.035001 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.035212 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.035320 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.037469 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:51.037479 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:51.037481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:51.037501 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:51.038698 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.039125 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.039129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.039131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.039209 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.039479 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.039483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.039485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.039643 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:51.039790 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.040131 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.040243 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:40:51.043334 2950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:40:51.046240 2950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:40:51.059799 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.053893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:46145 (local address 127.0.0.1:58704)
0928 14:40:51.054510 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.054556 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.054697 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.055940 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.055943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.055979 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:51.056211 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.056219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.057258 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.057265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.058437 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.058446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.059222 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.059225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.059327 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.059371 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.059517 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.059654 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":55,"spinlock_wait_cycles":3712,"thread_start_us":77,"threads_started":1}
I20260928 14:40:51.059806 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.054074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.1.201.2:46145)
0928 14:40:51.055523 (+ 1449us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.055536 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.055561 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:40:51.055640 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.055644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.055793 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.056332 (+ 539us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:51.056449 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.057126 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.058549 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.059073 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.059196 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.059359 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.059545 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.059667 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":110,"threads_started":1}
I20260928 14:40:51.060305 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.054088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:36033 (local address 127.0.0.1:52428)
0928 14:40:51.055193 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.055211 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.055341 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.055937 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.055943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.055977 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:51.056187 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.056193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.057683 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.057687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.058694 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.058703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.059888 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.059893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.059997 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.060017 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.060079 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.060141 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":127,"spinlock_wait_cycles":18560,"thread_start_us":110,"threads_started":1}
I20260928 14:40:51.060711 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.054637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52428 (local address 127.1.201.3:36033)
0928 14:40:51.055046 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.055059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.055459 (+ 400us) server_negotiation.cc:407] Connection header received
0928 14:40:51.055565 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.055570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.055780 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.055945 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:51.056304 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.057550 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.058824 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.059593 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.059688 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.059827 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.060495 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.060593 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":112,"threads_started":1}
I20260928 14:40:51.062362 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.054525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:33483 (local address 127.0.0.1:34568)
0928 14:40:51.055204 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.055215 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.055335 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.056044 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.056047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.056062 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:51.056215 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.056219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.057683 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.057687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.060149 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.060158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.061981 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.061985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.062071 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.062095 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.062174 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.062240 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":967,"spinlock_wait_cycles":2432,"thread_start_us":95,"threads_started":1}
I20260928 14:40:51.062429 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.054694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.1.201.1:33483)
0928 14:40:51.055012 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.055031 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.055308 (+ 277us) server_negotiation.cc:407] Connection header received
0928 14:40:51.055492 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.055498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.055744 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.055873 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:51.056358 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.057523 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.060262 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.061873 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.061981 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.062120 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.062208 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.062295 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":121,"threads_started":1}
I20260928 14:40:51.063186 3373 tablet_service.cc:1467] Processing CreateTablet for tablet 4a35551d61f04b928216ede635f87a06 (DEFAULT_TABLE table=test-table [id=2eef0f49714348a48c8dffca0d175d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:51.063060 3232 tablet_service.cc:1467] Processing CreateTablet for tablet 4a35551d61f04b928216ede635f87a06 (DEFAULT_TABLE table=test-table [id=2eef0f49714348a48c8dffca0d175d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:51.064031 3232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a35551d61f04b928216ede635f87a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:51.064040 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a35551d61f04b928216ede635f87a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:51.067891 3090 tablet_service.cc:1467] Processing CreateTablet for tablet 4a35551d61f04b928216ede635f87a06 (DEFAULT_TABLE table=test-table [id=2eef0f49714348a48c8dffca0d175d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:51.068812 3090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a35551d61f04b928216ede635f87a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:51.070797 3463 tablet_bootstrap.cc:492] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: Bootstrap starting.
I20260928 14:40:51.071702 3464 tablet_bootstrap.cc:492] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: Bootstrap starting.
I20260928 14:40:51.073091 3463 tablet_bootstrap.cc:654] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:51.073691 3464 tablet_bootstrap.cc:654] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:51.073989 3463 log.cc:824] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:51.074568 3464 log.cc:824] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:51.076345 3463 tablet_bootstrap.cc:492] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: No bootstrap required, opened a new log
I20260928 14:40:51.076526 3464 tablet_bootstrap.cc:492] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: No bootstrap required, opened a new log
I20260928 14:40:51.076532 3463 ts_tablet_manager.cc:1397] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:40:51.076543 3467 tablet_bootstrap.cc:492] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: Bootstrap starting.
I20260928 14:40:51.076730 3464 ts_tablet_manager.cc:1397] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:40:51.078648 3467 tablet_bootstrap.cc:654] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:51.079383 3467 log.cc:824] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:51.079993 3464 raft_consensus.cc:348] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } }
I20260928 14:40:51.080265 3464 raft_consensus.cc:374] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:51.080319 3464 raft_consensus.cc:729] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c79ccee5f24f9a9cffc9f18de6c55a, State: Initialized, Role: FOLLOWER
I20260928 14:40:51.080263 3463 raft_consensus.cc:348] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } }
I20260928 14:40:51.080482 3463 raft_consensus.cc:374] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:51.080533 3463 raft_consensus.cc:729] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc694b2a34d44f4b63b40790f3741cd, State: Initialized, Role: FOLLOWER
I20260928 14:40:51.080780 3464 consensus_queue.cc:260] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } }
I20260928 14:40:51.080914 3463 consensus_queue.cc:260] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } }
I20260928 14:40:51.081188 3467 tablet_bootstrap.cc:492] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: No bootstrap required, opened a new log
I20260928 14:40:51.081355 3467 ts_tablet_manager.cc:1397] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:40:51.081593 3440 heartbeater.cc:503] Master 127.1.201.62:44903 was elected leader, sending a full tablet report...
I20260928 14:40:51.081723 3464 ts_tablet_manager.cc:1428] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:40:51.081810 3463 ts_tablet_manager.cc:1428] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260928 14:40:51.082022 3299 heartbeater.cc:503] Master 127.1.201.62:44903 was elected leader, sending a full tablet report...
I20260928 14:40:51.085108 3467 raft_consensus.cc:348] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } }
I20260928 14:40:51.085333 3467 raft_consensus.cc:374] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:51.085371 3467 raft_consensus.cc:729] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0671218a17394bfb9285762f316b5606, State: Initialized, Role: FOLLOWER
I20260928 14:40:51.085783 3467 consensus_queue.cc:260] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } }
I20260928 14:40:51.086356 3157 heartbeater.cc:503] Master 127.1.201.62:44903 was elected leader, sending a full tablet report...
I20260928 14:40:51.086670 3467 ts_tablet_manager.cc:1428] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 14:40:51.101917 3158 tablet.cc:2369] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:40:51.258630 3441 tablet.cc:2369] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:40:51.305902 3300 tablet.cc:2369] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:40:51.346382 3471 raft_consensus.cc:482] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:40:51.346586 3471 raft_consensus.cc:504] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } }
I20260928 14:40:51.347671 3471 leader_election.cc:288] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14c79ccee5f24f9a9cffc9f18de6c55a (127.1.201.2:46145), 5bc694b2a34d44f4b63b40790f3741cd (127.1.201.3:36033)
I20260928 14:40:51.351725 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.348110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:58711 (local address 127.1.201.2:46145)
0928 14:40:51.348232 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.348236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.348249 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:40:51.348347 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.348350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.348420 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.348491 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:51.348761 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.349273 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.350735 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.351211 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.351325 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.351380 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.351533 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.351594 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:40:51.351756 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.347821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:46145 (local address 127.1.201.1:58711)
0928 14:40:51.348145 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.348160 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.348255 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.348515 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.348517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.348532 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:51.348646 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.348651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.349378 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.349383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.350600 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.350610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.351326 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.351329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.351410 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.351433 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.351519 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.351587 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":48,"thread_start_us":73,"threads_started":1}
I20260928 14:40:51.351827 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.348104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:42603 (local address 127.1.201.3:36033)
0928 14:40:51.348241 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.348247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.348261 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:40:51.348435 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.348440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.348497 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.348559 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:51.348868 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.349600 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.350904 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.351287 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.351393 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.351451 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.351615 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.351698 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:40:51.352028 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.347895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:36033 (local address 127.1.201.1:42603)
0928 14:40:51.348188 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.348202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.348290 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.348618 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.348622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.348642 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:51.348761 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.348767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.349717 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.349721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.350786 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.350792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.351395 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.351398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.351499 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.351523 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.351603 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.351694 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260928 14:40:51.352407 3253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a35551d61f04b928216ede635f87a06" candidate_uuid: "0671218a17394bfb9285762f316b5606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" is_pre_election: true
I20260928 14:40:51.352717 3253 raft_consensus.cc:2454] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0671218a17394bfb9285762f316b5606 in term 0.
I20260928 14:40:51.352720 3377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a35551d61f04b928216ede635f87a06" candidate_uuid: "0671218a17394bfb9285762f316b5606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc694b2a34d44f4b63b40790f3741cd" is_pre_election: true
I20260928 14:40:51.352950 3377 raft_consensus.cc:2454] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0671218a17394bfb9285762f316b5606 in term 0.
I20260928 14:40:51.353145 3043 leader_election.cc:302] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0671218a17394bfb9285762f316b5606, 14c79ccee5f24f9a9cffc9f18de6c55a; no voters:
I20260928 14:40:51.353374 3471 raft_consensus.cc:2781] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:40:51.353446 3471 raft_consensus.cc:482] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:40:51.353484 3471 raft_consensus.cc:3037] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:51.354426 3471 raft_consensus.cc:504] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } }
I20260928 14:40:51.354708 3471 leader_election.cc:288] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [CANDIDATE]: Term 1 election: Requested vote from peers 14c79ccee5f24f9a9cffc9f18de6c55a (127.1.201.2:46145), 5bc694b2a34d44f4b63b40790f3741cd (127.1.201.3:36033)
I20260928 14:40:51.354976 3253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a35551d61f04b928216ede635f87a06" candidate_uuid: "0671218a17394bfb9285762f316b5606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a"
I20260928 14:40:51.355024 3377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a35551d61f04b928216ede635f87a06" candidate_uuid: "0671218a17394bfb9285762f316b5606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc694b2a34d44f4b63b40790f3741cd"
I20260928 14:40:51.355079 3253 raft_consensus.cc:3037] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:51.355149 3377 raft_consensus.cc:3037] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:51.356073 3253 raft_consensus.cc:2454] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0671218a17394bfb9285762f316b5606 in term 1.
I20260928 14:40:51.356365 3043 leader_election.cc:302] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0671218a17394bfb9285762f316b5606, 14c79ccee5f24f9a9cffc9f18de6c55a; no voters:
I20260928 14:40:51.356534 3471 raft_consensus.cc:2781] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:51.356703 3471 raft_consensus.cc:686] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 1 LEADER]: Becoming Leader. State: Replica: 0671218a17394bfb9285762f316b5606, State: Running, Role: LEADER
I20260928 14:40:51.357033 3377 raft_consensus.cc:2454] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0671218a17394bfb9285762f316b5606 in term 1.
I20260928 14:40:51.357009 3471 consensus_queue.cc:237] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } }
I20260928 14:40:51.359763 2950 catalog_manager.cc:5495] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0671218a17394bfb9285762f316b5606 (127.1.201.1). New cstate: current_term: 1 leader_uuid: "0671218a17394bfb9285762f316b5606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } health_report { overall_health: HEALTHY } } }
I20260928 14:40:51.421891 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.418606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.1.201.1:33483)
0928 14:40:51.418726 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.418729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.418769 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:40:51.418951 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.418954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.419003 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.419081 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:51.419300 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.419772 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.420563 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.420748 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.420817 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.421059 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:51.421408 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:51.421518 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.421587 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.421662 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:40:51.427217 3253 raft_consensus.cc:1261] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a [term 1 FOLLOWER]: Refusing update from remote peer 0671218a17394bfb9285762f316b5606: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:51.427217 3377 raft_consensus.cc:1261] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd [term 1 FOLLOWER]: Refusing update from remote peer 0671218a17394bfb9285762f316b5606: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:51.427927 3474 consensus_queue.cc:1035] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:51.428112 3471 consensus_queue.cc:1035] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:51.434651 3478 mvcc.cc:205] Tried to move back new op lower bound from 7334324025038229504 to 7334324024760655872. Current Snapshot: MvccSnapshot[applied={T|T < 7334324025038229504}]
I20260928 14:40:51.444777 2951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:51.444890 2951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:51.447005 2951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161: Sending DeleteTablet for 3 replicas of tablet 4a35551d61f04b928216ede635f87a06
I20260928 14:40:51.447629 3232 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a35551d61f04b928216ede635f87a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:51 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:58704
I20260928 14:40:51.447633 3373 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a35551d61f04b928216ede635f87a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:51 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:52428
I20260928 14:40:51.447810 3090 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a35551d61f04b928216ede635f87a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:51 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:34568
I20260928 14:40:51.448033 3486 tablet_replica.cc:330] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: stopping tablet replica
I20260928 14:40:51.448035 3487 tablet_replica.cc:330] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: stopping tablet replica
I20260928 14:40:51.448289 3487 raft_consensus.cc:2229] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:51.448289 3486 raft_consensus.cc:2229] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:51.448540 3486 raft_consensus.cc:2258] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:51.448540 3487 raft_consensus.cc:2258] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:51.448676 2951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:40:51.449036 2951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:40:51.449747 3487 ts_tablet_manager.cc:1905] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:51.448894 3488 tablet_replica.cc:330] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: stopping tablet replica
I20260928 14:40:51.450201 3488 raft_consensus.cc:2229] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:40:51.450495 3488 raft_consensus.cc:2258] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:51.451061 3486 ts_tablet_manager.cc:1905] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:51.452497 3487 ts_tablet_manager.cc:1918] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:40:51.452582 3487 log.cc:1196] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/4a35551d61f04b928216ede635f87a06
I20260928 14:40:51.452885 3487 ts_tablet_manager.cc:1939] T 4a35551d61f04b928216ede635f87a06 P 5bc694b2a34d44f4b63b40790f3741cd: Deleting consensus metadata
I20260928 14:40:51.453462 3486 ts_tablet_manager.cc:1918] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:40:51.453543 3486 log.cc:1196] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/4a35551d61f04b928216ede635f87a06
I20260928 14:40:51.453835 3486 ts_tablet_manager.cc:1939] T 4a35551d61f04b928216ede635f87a06 P 14c79ccee5f24f9a9cffc9f18de6c55a: Deleting consensus metadata
I20260928 14:40:51.454077 3232 tablet_service.cc:1467] Processing CreateTablet for tablet 80846f3be524476a898ed33f21293f8e (DEFAULT_TABLE table=test-table [id=557c24a07f084a0ea4d9cd003c5cd4aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:51.454308 3232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80846f3be524476a898ed33f21293f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:51.454413 3373 tablet_service.cc:1467] Processing CreateTablet for tablet 80846f3be524476a898ed33f21293f8e (DEFAULT_TABLE table=test-table [id=557c24a07f084a0ea4d9cd003c5cd4aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:51.454419 3090 tablet_service.cc:1467] Processing CreateTablet for tablet 80846f3be524476a898ed33f21293f8e (DEFAULT_TABLE table=test-table [id=557c24a07f084a0ea4d9cd003c5cd4aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:51.454614 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80846f3be524476a898ed33f21293f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:51.454631 3090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80846f3be524476a898ed33f21293f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:51.456612 2937 catalog_manager.cc:4841] TS 14c79ccee5f24f9a9cffc9f18de6c55a (127.1.201.2:46145): tablet 4a35551d61f04b928216ede635f87a06 (table test-table [id=2eef0f49714348a48c8dffca0d175d8f]) successfully deleted
I20260928 14:40:51.456697 3463 tablet_bootstrap.cc:492] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd: Bootstrap starting.
I20260928 14:40:51.457089 3464 tablet_bootstrap.cc:492] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a: Bootstrap starting.
I20260928 14:40:51.457564 3463 tablet_bootstrap.cc:654] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:51.457911 3464 tablet_bootstrap.cc:654] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:51.457935 3488 ts_tablet_manager.cc:1905] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:51.458315 2936 catalog_manager.cc:4841] TS 5bc694b2a34d44f4b63b40790f3741cd (127.1.201.3:36033): tablet 4a35551d61f04b928216ede635f87a06 (table test-table [id=2eef0f49714348a48c8dffca0d175d8f]) successfully deleted
I20260928 14:40:51.458542 3467 tablet_bootstrap.cc:492] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606: Bootstrap starting.
I20260928 14:40:51.459312 3464 tablet_bootstrap.cc:492] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a: No bootstrap required, opened a new log
I20260928 14:40:51.459401 3467 tablet_bootstrap.cc:654] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:51.459400 3464 ts_tablet_manager.cc:1397] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:51.459769 3464 raft_consensus.cc:348] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.459880 3464 raft_consensus.cc:374] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:51.459909 3464 raft_consensus.cc:729] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c79ccee5f24f9a9cffc9f18de6c55a, State: Initialized, Role: FOLLOWER
I20260928 14:40:51.460011 3464 consensus_queue.cc:260] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.460280 3464 ts_tablet_manager.cc:1428] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:51.460322 3488 ts_tablet_manager.cc:1918] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:40:51.460402 3488 log.cc:1196] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/4a35551d61f04b928216ede635f87a06
I20260928 14:40:51.460654 3463 tablet_bootstrap.cc:492] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd: No bootstrap required, opened a new log
I20260928 14:40:51.460682 3488 ts_tablet_manager.cc:1939] T 4a35551d61f04b928216ede635f87a06 P 0671218a17394bfb9285762f316b5606: Deleting consensus metadata
I20260928 14:40:51.460739 3463 ts_tablet_manager.cc:1397] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:40:51.461109 3463 raft_consensus.cc:348] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.461202 3463 raft_consensus.cc:374] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:51.461237 3463 raft_consensus.cc:729] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc694b2a34d44f4b63b40790f3741cd, State: Initialized, Role: FOLLOWER
I20260928 14:40:51.461376 3463 consensus_queue.cc:260] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.461591 3463 ts_tablet_manager.cc:1428] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:51.462163 2935 catalog_manager.cc:4841] TS 0671218a17394bfb9285762f316b5606 (127.1.201.1:33483): tablet 4a35551d61f04b928216ede635f87a06 (table test-table [id=2eef0f49714348a48c8dffca0d175d8f]) successfully deleted
I20260928 14:40:51.462201 3467 tablet_bootstrap.cc:492] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606: No bootstrap required, opened a new log
I20260928 14:40:51.462275 3467 ts_tablet_manager.cc:1397] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 14:40:51.462677 3467 raft_consensus.cc:348] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.462786 3467 raft_consensus.cc:374] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:51.462824 3467 raft_consensus.cc:729] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0671218a17394bfb9285762f316b5606, State: Initialized, Role: FOLLOWER
I20260928 14:40:51.462937 3467 consensus_queue.cc:260] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.463172 3467 ts_tablet_manager.cc:1428] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:40:51.475165 3471 raft_consensus.cc:482] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:40:51.475244 3471 raft_consensus.cc:504] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.475533 3471 leader_election.cc:288] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14c79ccee5f24f9a9cffc9f18de6c55a (127.1.201.2:46145), 5bc694b2a34d44f4b63b40790f3741cd (127.1.201.3:36033)
I20260928 14:40:51.475829 3377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80846f3be524476a898ed33f21293f8e" candidate_uuid: "0671218a17394bfb9285762f316b5606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc694b2a34d44f4b63b40790f3741cd" is_pre_election: true
I20260928 14:40:51.475831 3253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80846f3be524476a898ed33f21293f8e" candidate_uuid: "0671218a17394bfb9285762f316b5606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" is_pre_election: true
I20260928 14:40:51.475940 3377 raft_consensus.cc:2454] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0671218a17394bfb9285762f316b5606 in term 0.
I20260928 14:40:51.475962 3253 raft_consensus.cc:2454] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0671218a17394bfb9285762f316b5606 in term 0.
I20260928 14:40:51.476212 3042 leader_election.cc:302] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0671218a17394bfb9285762f316b5606, 5bc694b2a34d44f4b63b40790f3741cd; no voters:
I20260928 14:40:51.476379 3471 raft_consensus.cc:2781] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:40:51.476428 3471 raft_consensus.cc:482] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:40:51.476462 3471 raft_consensus.cc:3037] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:51.477345 3471 raft_consensus.cc:504] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.477639 3471 leader_election.cc:288] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [CANDIDATE]: Term 1 election: Requested vote from peers 14c79ccee5f24f9a9cffc9f18de6c55a (127.1.201.2:46145), 5bc694b2a34d44f4b63b40790f3741cd (127.1.201.3:36033)
I20260928 14:40:51.478012 3253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80846f3be524476a898ed33f21293f8e" candidate_uuid: "0671218a17394bfb9285762f316b5606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a"
I20260928 14:40:51.477994 3377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80846f3be524476a898ed33f21293f8e" candidate_uuid: "0671218a17394bfb9285762f316b5606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc694b2a34d44f4b63b40790f3741cd"
I20260928 14:40:51.478096 3253 raft_consensus.cc:3037] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:51.478104 3377 raft_consensus.cc:3037] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:51.478997 3377 raft_consensus.cc:2454] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0671218a17394bfb9285762f316b5606 in term 1.
I20260928 14:40:51.478997 3253 raft_consensus.cc:2454] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0671218a17394bfb9285762f316b5606 in term 1.
I20260928 14:40:51.479235 3043 leader_election.cc:302] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0671218a17394bfb9285762f316b5606, 14c79ccee5f24f9a9cffc9f18de6c55a; no voters:
I20260928 14:40:51.479378 3471 raft_consensus.cc:2781] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:51.479471 3471 raft_consensus.cc:686] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 1 LEADER]: Becoming Leader. State: Replica: 0671218a17394bfb9285762f316b5606, State: Running, Role: LEADER
I20260928 14:40:51.479600 3471 consensus_queue.cc:237] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.480831 2950 catalog_manager.cc:5495] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0671218a17394bfb9285762f316b5606 (127.1.201.1). New cstate: current_term: 1 leader_uuid: "0671218a17394bfb9285762f316b5606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:40:51 dist-test-slave-0n27 krb5kdc[2905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606450, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:51.523213 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.515126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44903 (local address 127.0.0.1:53096)
0928 14:40:51.515409 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.515421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.515533 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.515802 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.515805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.516184 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:51.516263 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.516269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.516999 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.517004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.517536 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.517542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.518059 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.518062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.518140 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.518661 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:51.518671 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:51.520153 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:51.521612 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:51.521624 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:51.521629 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:51.521918 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:51.522191 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:51.522193 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:51.522195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:51.522270 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:51.522650 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:51.522658 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:51.522762 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.522869 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.523012 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
I20260928 14:40:51.523236 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.515171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.1.201.62:44903)
0928 14:40:51.515288 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.515293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.515492 (+ 199us) server_negotiation.cc:407] Connection header received
0928 14:40:51.515658 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.515661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.515698 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.515787 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:51.516376 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.516899 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.517648 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.517949 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.518051 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.520279 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:51.520282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:51.520285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:51.520311 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:51.521511 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.522009 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.522025 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.522027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.522096 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.522385 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.522387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.522389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.522550 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:51.522650 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.522922 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.523060 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":45}
I20260928 14:40:51.526916 2950 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:53096:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:40:51.527567 2950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:40:51.531989 3090 tablet_service.cc:1467] Processing CreateTablet for tablet fe7c3202a5174e5dbf8637a92134fdcb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ab1c2649cb146948421146baab31b06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:40:51.532012 3373 tablet_service.cc:1467] Processing CreateTablet for tablet fe7c3202a5174e5dbf8637a92134fdcb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ab1c2649cb146948421146baab31b06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:40:51.532083 3232 tablet_service.cc:1467] Processing CreateTablet for tablet fe7c3202a5174e5dbf8637a92134fdcb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ab1c2649cb146948421146baab31b06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:40:51.532313 3090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7c3202a5174e5dbf8637a92134fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:51.532357 3232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7c3202a5174e5dbf8637a92134fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:51.532357 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7c3202a5174e5dbf8637a92134fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:51.534785 3467 tablet_bootstrap.cc:492] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: Bootstrap starting.
I20260928 14:40:51.535005 3464 tablet_bootstrap.cc:492] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: Bootstrap starting.
I20260928 14:40:51.535655 3467 tablet_bootstrap.cc:654] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:51.535835 3464 tablet_bootstrap.cc:654] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:51.536271 3463 tablet_bootstrap.cc:492] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: Bootstrap starting.
I20260928 14:40:51.536908 3464 tablet_bootstrap.cc:492] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: No bootstrap required, opened a new log
I20260928 14:40:51.537009 3464 ts_tablet_manager.cc:1397] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:40:51.537269 3463 tablet_bootstrap.cc:654] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:51.537515 3464 raft_consensus.cc:348] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.537637 3464 raft_consensus.cc:374] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:51.537676 3464 raft_consensus.cc:729] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c79ccee5f24f9a9cffc9f18de6c55a, State: Initialized, Role: FOLLOWER
I20260928 14:40:51.537879 3464 consensus_queue.cc:260] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.538193 3464 ts_tablet_manager.cc:1428] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:51.538412 3469 tablet_replica.cc:439] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } }
I20260928 14:40:51.538512 3467 tablet_bootstrap.cc:492] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: No bootstrap required, opened a new log
I20260928 14:40:51.538534 3469 tablet_replica.cc:442] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:51.538586 3467 ts_tablet_manager.cc:1397] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 14:40:51.538977 3467 raft_consensus.cc:348] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.539078 3467 raft_consensus.cc:374] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:51.539112 3467 raft_consensus.cc:729] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0671218a17394bfb9285762f316b5606, State: Initialized, Role: FOLLOWER
I20260928 14:40:51.539218 3467 consensus_queue.cc:260] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.539463 3467 ts_tablet_manager.cc:1428] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:51.539621 3471 tablet_replica.cc:439] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } }
I20260928 14:40:51.539731 3471 tablet_replica.cc:442] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:51.540551 3463 tablet_bootstrap.cc:492] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: No bootstrap required, opened a new log
I20260928 14:40:51.540639 3463 ts_tablet_manager.cc:1397] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:40:51.541038 3463 raft_consensus.cc:348] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.541141 3463 raft_consensus.cc:374] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:51.541177 3463 raft_consensus.cc:729] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc694b2a34d44f4b63b40790f3741cd, State: Initialized, Role: FOLLOWER
I20260928 14:40:51.541290 3463 consensus_queue.cc:260] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.541544 3463 ts_tablet_manager.cc:1428] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:51.541716 3470 tablet_replica.cc:439] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } }
I20260928 14:40:51.541865 3470 tablet_replica.cc:442] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:51.550419 3470 raft_consensus.cc:482] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:40:51.550518 3470 raft_consensus.cc:504] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.551576 3470 leader_election.cc:288] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0671218a17394bfb9285762f316b5606 (127.1.201.1:33483), 14c79ccee5f24f9a9cffc9f18de6c55a (127.1.201.2:46145)
I20260928 14:40:51.556218 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.551870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:46847 (local address 127.1.201.1:33483)
0928 14:40:51.551970 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.551974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.552556 (+ 582us) server_negotiation.cc:407] Connection header received
0928 14:40:51.552658 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.552661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.552704 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.552779 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:51.553055 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.553756 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.555265 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.555654 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.555777 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.555837 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.556055 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.556129 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:40:51.556506 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.552059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:57563 (local address 127.1.201.2:46145)
0928 14:40:51.552161 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.552167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.552183 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:40:51.552342 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.552345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.552390 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.552464 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:51.552839 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.553583 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.555246 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.555857 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.555960 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.556013 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.556317 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.556404 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:40:51.556819 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.551754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:46145 (local address 127.1.201.3:57563)
0928 14:40:51.552110 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.552125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.552227 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.552572 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.552577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.552598 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:51.552716 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.552724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.553719 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.553722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.555124 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.555134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.556069 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.556073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.556174 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.556199 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.556429 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.556516 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":66,"thread_start_us":81,"threads_started":1}
I20260928 14:40:51.556838 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.551754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:33483 (local address 127.1.201.3:46847)
0928 14:40:51.552494 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.552506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.552581 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.552805 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.552808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.552823 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:51.552949 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.552955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.553895 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.553898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.555124 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.555134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.555778 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.555781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.555906 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.555939 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.556399 (+ 460us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.556723 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":58,"thread_start_us":67,"threads_started":1}
I20260928 14:40:51.557364 3110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7c3202a5174e5dbf8637a92134fdcb" candidate_uuid: "5bc694b2a34d44f4b63b40790f3741cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0671218a17394bfb9285762f316b5606" is_pre_election: true
I20260928 14:40:51.557426 3253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7c3202a5174e5dbf8637a92134fdcb" candidate_uuid: "5bc694b2a34d44f4b63b40790f3741cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" is_pre_election: true
I20260928 14:40:51.557523 3253 raft_consensus.cc:2454] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bc694b2a34d44f4b63b40790f3741cd in term 0.
I20260928 14:40:51.557554 3110 raft_consensus.cc:2454] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bc694b2a34d44f4b63b40790f3741cd in term 0.
I20260928 14:40:51.557924 3326 leader_election.cc:302] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0671218a17394bfb9285762f316b5606, 14c79ccee5f24f9a9cffc9f18de6c55a, 5bc694b2a34d44f4b63b40790f3741cd; no voters:
I20260928 14:40:51.558153 3470 raft_consensus.cc:2781] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:40:51.558214 3470 raft_consensus.cc:482] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:40:51.558254 3470 raft_consensus.cc:3037] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:51.559346 3470 raft_consensus.cc:504] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.559664 3470 leader_election.cc:288] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [CANDIDATE]: Term 1 election: Requested vote from peers 0671218a17394bfb9285762f316b5606 (127.1.201.1:33483), 14c79ccee5f24f9a9cffc9f18de6c55a (127.1.201.2:46145)
I20260928 14:40:51.560280 3110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7c3202a5174e5dbf8637a92134fdcb" candidate_uuid: "5bc694b2a34d44f4b63b40790f3741cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0671218a17394bfb9285762f316b5606"
I20260928 14:40:51.560449 3110 raft_consensus.cc:3037] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:51.562671 3110 raft_consensus.cc:2454] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bc694b2a34d44f4b63b40790f3741cd in term 1.
I20260928 14:40:51.563081 3324 leader_election.cc:302] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0671218a17394bfb9285762f316b5606, 5bc694b2a34d44f4b63b40790f3741cd; no voters:
I20260928 14:40:51.563298 3470 raft_consensus.cc:2781] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:51.563359 3253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7c3202a5174e5dbf8637a92134fdcb" candidate_uuid: "5bc694b2a34d44f4b63b40790f3741cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a"
I20260928 14:40:51.563458 3253 raft_consensus.cc:3037] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:51.563452 3470 raft_consensus.cc:686] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [term 1 LEADER]: Becoming Leader. State: Replica: 5bc694b2a34d44f4b63b40790f3741cd, State: Running, Role: LEADER
I20260928 14:40:51.563722 3470 consensus_queue.cc:237] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } }
I20260928 14:40:51.564463 3253 raft_consensus.cc:2454] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bc694b2a34d44f4b63b40790f3741cd in term 1.
I20260928 14:40:51.564994 3504 tablet_replica.cc:439] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: TxnStatusTablet state changed. Reason: New leader 5bc694b2a34d44f4b63b40790f3741cd. Latest consensus state: current_term: 1 leader_uuid: "5bc694b2a34d44f4b63b40790f3741cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } }
I20260928 14:40:51.565102 3504 tablet_replica.cc:442] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:40:51.565441 3506 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:40:51.565682 3506 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:40:51.566565 2950 catalog_manager.cc:5495] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bc694b2a34d44f4b63b40790f3741cd (127.1.201.3). New cstate: current_term: 1 leader_uuid: "5bc694b2a34d44f4b63b40790f3741cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } health_report { overall_health: HEALTHY } } }
I20260928 14:40:51.584105 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.579657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:36033 (local address 127.0.0.1:52436)
0928 14:40:51.579780 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.579795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.579884 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.580292 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.580295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.580304 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:40:51.580397 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.580402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.581364 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.581367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.582335 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.582346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.582971 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.582974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.583053 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.583103 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:40:51.583821 (+ 718us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:40:51.583834 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.583892 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.583969 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 14:40:51.584545 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.579770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.1.201.3:36033)
0928 14:40:51.579892 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.579896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.579909 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:40:51.580042 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.580045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.580101 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.580176 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:51.580504 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.581231 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.582467 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.582810 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.582894 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.583248 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:51.583686 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:51.583779 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.584376 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.584432 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:40:51.587009 3253 raft_consensus.cc:1261] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a [term 1 FOLLOWER]: Refusing update from remote peer 5bc694b2a34d44f4b63b40790f3741cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:51.587385 3110 raft_consensus.cc:1261] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606 [term 1 FOLLOWER]: Refusing update from remote peer 5bc694b2a34d44f4b63b40790f3741cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:51.587487 3470 consensus_queue.cc:1035] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:51.587872 3470 consensus_queue.cc:1035] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:51.588881 3469 tablet_replica.cc:439] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: TxnStatusTablet state changed. Reason: New leader 5bc694b2a34d44f4b63b40790f3741cd. Latest consensus state: current_term: 1 leader_uuid: "5bc694b2a34d44f4b63b40790f3741cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } }
I20260928 14:40:51.588977 3469 tablet_replica.cc:442] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:51.589447 3471 tablet_replica.cc:439] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: TxnStatusTablet state changed. Reason: New leader 5bc694b2a34d44f4b63b40790f3741cd. Latest consensus state: current_term: 1 leader_uuid: "5bc694b2a34d44f4b63b40790f3741cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } }
I20260928 14:40:51.589545 3471 tablet_replica.cc:442] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:51.591276 3470 tablet_replica.cc:439] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bc694b2a34d44f4b63b40790f3741cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } }
I20260928 14:40:51.591374 3470 tablet_replica.cc:442] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:40:51.591480 3470 tablet_replica.cc:439] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bc694b2a34d44f4b63b40790f3741cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } }
I20260928 14:40:51.591545 3470 tablet_replica.cc:442] T fe7c3202a5174e5dbf8637a92134fdcb P 5bc694b2a34d44f4b63b40790f3741cd: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:40:51.592703 3479 mvcc.cc:205] Tried to move back new op lower bound from 7334324025696251904 to 7334324025608712192. Current Snapshot: MvccSnapshot[applied={T|T < 7334324025696251904}]
I20260928 14:40:51.592872 3469 tablet_replica.cc:439] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bc694b2a34d44f4b63b40790f3741cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } }
I20260928 14:40:51.592952 3469 tablet_replica.cc:442] T fe7c3202a5174e5dbf8637a92134fdcb P 14c79ccee5f24f9a9cffc9f18de6c55a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:51.593880 3471 tablet_replica.cc:439] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bc694b2a34d44f4b63b40790f3741cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0671218a17394bfb9285762f316b5606" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33483 } } peers { permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 } } peers { permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 } } }
I20260928 14:40:51.594005 3471 tablet_replica.cc:442] T fe7c3202a5174e5dbf8637a92134fdcb P 0671218a17394bfb9285762f316b5606: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:51.612881 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.608584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:45779 (local address 127.1.201.3:36033)
0928 14:40:51.608678 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.608682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.608977 (+ 295us) server_negotiation.cc:407] Connection header received
0928 14:40:51.609153 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.609156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.609199 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.609240 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:51.609560 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.610298 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.611589 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.611896 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.611957 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.612356 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:51.612449 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:51.612495 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.612679 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.612739 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:40:51.613302 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.608264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:36033 (local address 127.1.201.1:45779)
0928 14:40:51.608889 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.608902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.609041 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.609353 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.609367 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.609374 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:40:51.609460 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.609464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.610454 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.610457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.611464 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.611474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.612051 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.612054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.612138 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.612188 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:40:51.612555 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:40:51.612562 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.613045 (+ 483us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.613146 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":86,"threads_started":1}
I20260928 14:40:51.621486 3252 raft_consensus.cc:1261] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a [term 1 FOLLOWER]: Refusing update from remote peer 0671218a17394bfb9285762f316b5606: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:51.621840 3377 raft_consensus.cc:1261] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd [term 1 FOLLOWER]: Refusing update from remote peer 0671218a17394bfb9285762f316b5606: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:51.622198 3474 consensus_queue.cc:1035] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c79ccee5f24f9a9cffc9f18de6c55a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:51.622612 3474 consensus_queue.cc:1035] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc694b2a34d44f4b63b40790f3741cd" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:51.627478 3477 mvcc.cc:205] Tried to move back new op lower bound from 7334324025837527040 to 7334324025261694976. Current Snapshot: MvccSnapshot[applied={T|T < 7334324025837527040}]
I20260928 14:40:51.644184 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.640598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:58261 (local address 127.1.201.1:33483)
0928 14:40:51.640716 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.640721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.640885 (+ 164us) server_negotiation.cc:407] Connection header received
0928 14:40:51.640963 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.640966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.641010 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.641096 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:51.641291 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.642026 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.643201 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.643415 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.643486 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.643838 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:51.643896 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:51.643947 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.644065 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.644104 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:40:51.644274 3518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.640493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:33483 (local address 127.1.201.3:58261)
0928 14:40:51.640809 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:51.640821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:51.640899 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:51.641099 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:51.641101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:51.641105 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:40:51.641206 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.641210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.642144 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.642147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.643083 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:51.643093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.643517 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.643521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:51.643627 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.643673 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:40:51.643979 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:40:51.643985 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:40:51.644049 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:40:51.644120 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260928 14:40:51.662997 2949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:51.663105 2949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:51.664887 2949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f35ca77aa7a48d491b3f77d29ba5161: Sending DeleteTablet for 3 replicas of tablet 80846f3be524476a898ed33f21293f8e
I20260928 14:40:51.665424 3232 tablet_service.cc:1514] Processing DeleteTablet for tablet 80846f3be524476a898ed33f21293f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:51 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:58704
I20260928 14:40:51.665513 3373 tablet_service.cc:1514] Processing DeleteTablet for tablet 80846f3be524476a898ed33f21293f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:51 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:52428
I20260928 14:40:51.665599 3486 tablet_replica.cc:330] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a: stopping tablet replica
I20260928 14:40:51.665664 3487 tablet_replica.cc:330] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd: stopping tablet replica
I20260928 14:40:51.665726 3486 raft_consensus.cc:2229] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:51.665762 3487 raft_consensus.cc:2229] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:51.665869 3487 raft_consensus.cc:2258] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:51.665864 3486 raft_consensus.cc:2258] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:51.666121 3090 tablet_service.cc:1514] Processing DeleteTablet for tablet 80846f3be524476a898ed33f21293f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:51 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:34568
I20260928 14:40:51.666285 3488 tablet_replica.cc:330] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606: stopping tablet replica
I20260928 14:40:51.666394 3488 raft_consensus.cc:2229] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:40:51.666549 3488 raft_consensus.cc:2258] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:51.666580 3487 ts_tablet_manager.cc:1905] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:51.668000 3488 ts_tablet_manager.cc:1905] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:51.669160 3487 ts_tablet_manager.cc:1918] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:40:51.669234 3487 log.cc:1196] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/80846f3be524476a898ed33f21293f8e
I20260928 14:40:51.669484 3487 ts_tablet_manager.cc:1939] T 80846f3be524476a898ed33f21293f8e P 5bc694b2a34d44f4b63b40790f3741cd: Deleting consensus metadata
I20260928 14:40:51.670066 2936 catalog_manager.cc:4841] TS 5bc694b2a34d44f4b63b40790f3741cd (127.1.201.3:36033): tablet 80846f3be524476a898ed33f21293f8e (table test-table [id=557c24a07f084a0ea4d9cd003c5cd4aa]) successfully deleted
I20260928 14:40:51.670487 3488 ts_tablet_manager.cc:1918] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:40:51.670543 3488 log.cc:1196] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/80846f3be524476a898ed33f21293f8e
I20260928 14:40:51.670775 3488 ts_tablet_manager.cc:1939] T 80846f3be524476a898ed33f21293f8e P 0671218a17394bfb9285762f316b5606: Deleting consensus metadata
I20260928 14:40:51.671399 2935 catalog_manager.cc:4841] TS 0671218a17394bfb9285762f316b5606 (127.1.201.1:33483): tablet 80846f3be524476a898ed33f21293f8e (table test-table [id=557c24a07f084a0ea4d9cd003c5cd4aa]) successfully deleted
I20260928 14:40:51.671514 3486 ts_tablet_manager.cc:1905] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:51.674135 3486 ts_tablet_manager.cc:1918] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:40:51.674208 3486 log.cc:1196] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/80846f3be524476a898ed33f21293f8e
I20260928 14:40:51.674474 3486 ts_tablet_manager.cc:1939] T 80846f3be524476a898ed33f21293f8e P 14c79ccee5f24f9a9cffc9f18de6c55a: Deleting consensus metadata
I20260928 14:40:51.675006 2937 catalog_manager.cc:4841] TS 14c79ccee5f24f9a9cffc9f18de6c55a (127.1.201.2:46145): tablet 80846f3be524476a898ed33f21293f8e (table test-table [id=557c24a07f084a0ea4d9cd003c5cd4aa]) successfully deleted
Sep 28 14:40:51 dist-test-slave-0n27 krb5kdc[2905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:40:51.681280 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.670976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.1.201.1:33483)
0928 14:40:51.671080 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.671084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.671521 (+ 437us) server_negotiation.cc:407] Connection header received
0928 14:40:51.671671 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.671674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.671718 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.671791 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:51.672542 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.673278 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.674537 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.674863 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.675071 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.678344 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:51.678352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:51.678362 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:51.678380 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:51.679826 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.680247 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.680251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.680253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.680327 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.680570 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.680572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.680573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.680766 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:51.680907 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.681106 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.681182 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 14:40:51.681710 3150 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34592
I20260928 14:40:51.690256 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:51.683205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.1.201.62:44903)
0928 14:40:51.683325 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:51.683330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:51.683554 (+ 224us) server_negotiation.cc:407] Connection header received
0928 14:40:51.683637 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:51.683640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:51.683679 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:51.683745 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:51.684226 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.684930 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.685591 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:51.685973 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:51.686069 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:51.687196 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:51.687199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:51.687202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:51.687224 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:51.688554 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.688974 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.688991 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.688995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.689076 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:51.689376 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:51.689379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:51.689381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:51.689575 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:51.689713 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:51.689998 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 14:40:51.690119 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 14:40:51.690603 2949 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53106
I20260928 14:40:51.691699 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 3024
I20260928 14:40:51.700989 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 3166
I20260928 14:40:51.707206 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 3308
I20260928 14:40:51.712131 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 2918
2026-09-28T14:40:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2615 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:51 dist-test-slave-0n27 krb5kdc[3527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:51 dist-test-slave-0n27 krb5kdc[3527](info): set up 2 sockets
Sep 28 14:40:51 dist-test-slave-0n27 krb5kdc[3527](info): commencing operation
krb5kdc: starting...
W20260928 14:40:52.694381 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:40:52 dist-test-slave-0n27 krb5kdc[3527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:40:52.806571 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:39223
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:39223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:52.908623 3541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:52.908840 3541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:52.908876 3541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:52.912745 3541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:52.912791 3541 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:40:52.912807 3541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:52.912824 3541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:52.912839 3541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:52.912854 3541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:52.917912 3541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:39223
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:39223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:52.918764 3541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:52.919667 3541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:40:52.927662 3541 server_base.cc:1034] running on GCE node
W20260928 14:40:52.927615 3546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:52.927620 3547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:52.927615 3549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:52.928292 3541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:52.929258 3541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:52.930409 3541 hybrid_clock.cc:648] HybridClock initialized: now 1790606452930384 us; error 36 us; skew 500 ppm
Sep 28 14:40:52 dist-test-slave-0n27 krb5kdc[3527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:52.932766 3541 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:40:52.933534 3541 webserver.cc:460] Webserver started at http://127.1.201.62:33983/ using document root <none> and password file <none>
I20260928 14:40:52.934087 3541 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:52.934168 3541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:52.934384 3541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:52.936187 3541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "05f83e020b0c4da0b3ce6332360705c4"
format_stamp: "Formatted at 2026-09-28 14:40:52 on dist-test-slave-0n27"
server_key: "771682b21c3ba9e16b15e339bca8b339"
server_key_iv: "2b503f0cdef065815a508a7b2292b3cf"
server_key_version: "encryptionkey@0"
I20260928 14:40:52.936712 3541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "05f83e020b0c4da0b3ce6332360705c4"
format_stamp: "Formatted at 2026-09-28 14:40:52 on dist-test-slave-0n27"
server_key: "771682b21c3ba9e16b15e339bca8b339"
server_key_iv: "2b503f0cdef065815a508a7b2292b3cf"
server_key_version: "encryptionkey@0"
I20260928 14:40:52.939986 3541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:40:52.942212 3556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:52.943324 3541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:52.943451 3541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "05f83e020b0c4da0b3ce6332360705c4"
format_stamp: "Formatted at 2026-09-28 14:40:52 on dist-test-slave-0n27"
server_key: "771682b21c3ba9e16b15e339bca8b339"
server_key_iv: "2b503f0cdef065815a508a7b2292b3cf"
server_key_version: "encryptionkey@0"
I20260928 14:40:52.943550 3541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:52.955962 3541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:52.958374 3541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:52.958535 3541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:52.965346 3541 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:39223
I20260928 14:40:52.965356 3618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:39223 every 8 connection(s)
I20260928 14:40:52.966213 3541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:52.968591 3619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:52.971155 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 3541
I20260928 14:40:52.971277 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:40:52.971554 1828 external_mini_cluster.cc:1428] Setting key 5d3ca898361183cb413fc91396829913
I20260928 14:40:52.973770 3619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4: Bootstrap starting.
I20260928 14:40:52.976162 3619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:52.976924 3619 log.cc:824] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:52.978665 3619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4: No bootstrap required, opened a new log
Sep 28 14:40:52 dist-test-slave-0n27 krb5kdc[3527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:52.982033 3619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f83e020b0c4da0b3ce6332360705c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39223 } }
I20260928 14:40:52.982218 3619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:52.982295 3619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f83e020b0c4da0b3ce6332360705c4, State: Initialized, Role: FOLLOWER
I20260928 14:40:52.982733 3619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f83e020b0c4da0b3ce6332360705c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39223 } }
I20260928 14:40:52.982842 3619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:52.982915 3619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:52.983006 3619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:52.983767 3619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f83e020b0c4da0b3ce6332360705c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39223 } }
I20260928 14:40:52.984087 3619 leader_election.cc:302] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f83e020b0c4da0b3ce6332360705c4; no voters:
I20260928 14:40:52.984431 3619 leader_election.cc:288] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:52.984495 3624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:52.984635 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:52.972730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.1.201.62:39223)
0928 14:40:52.973071 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:52.973087 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:52.973123 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:40:52.973845 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:52.973867 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:52.974225 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:52.974529 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:52.975075 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:52.976516 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:52.977395 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:52.977864 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:52.978016 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:52.980275 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:52.980280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:52.980290 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:52.980311 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:52.981759 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:52.982193 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:52.982215 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:52.982221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:52.982371 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:52.982693 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:52.982696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:52.982697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:52.983136 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:52.983321 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:52.983591 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 14:40:52.983841 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":81,"threads_started":1}
I20260928 14:40:52.984666 3624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [term 1 LEADER]: Becoming Leader. State: Replica: 05f83e020b0c4da0b3ce6332360705c4, State: Running, Role: LEADER
I20260928 14:40:52.984927 3624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f83e020b0c4da0b3ce6332360705c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39223 } }
I20260928 14:40:52.985513 3619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:52.987321 3626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05f83e020b0c4da0b3ce6332360705c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f83e020b0c4da0b3ce6332360705c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39223 } } }
I20260928 14:40:52.987486 3626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:52.987488 3625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05f83e020b0c4da0b3ce6332360705c4. Latest consensus state: current_term: 1 leader_uuid: "05f83e020b0c4da0b3ce6332360705c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f83e020b0c4da0b3ce6332360705c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39223 } } }
I20260928 14:40:52.987670 3625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4 [sys.catalog]: This master's current role is: LEADER
W20260928 14:40:52.989753 3639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:52.989836 3639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:52.989897 3640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:52.990511 3640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:52.995357 3640 catalog_manager.cc:1347] Generated new cluster ID: c29499059887406c9ad3b8274cc5bec9
I20260928 14:40:52.995414 3640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:53.009483 3640 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:53.010500 3640 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:53.020851 3640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4: Generated new TSK 0
I20260928 14:40:53.021436 3640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:40:53.048364 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39223
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:53.150230 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:53.150425 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:53.150491 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:53.154112 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:53.154151 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:53.154232 3645 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:40:53.154265 3645 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:40:53.154322 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:40:53.159089 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.201.62:39223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:53.159934 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:53.160753 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:53.169479 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:53.169479 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:53.169502 3645 server_base.cc:1034] running on GCE node
W20260928 14:40:53.169739 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:53.170116 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:53.170677 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:53.171847 3645 hybrid_clock.cc:648] HybridClock initialized: now 1790606453171817 us; error 42 us; skew 500 ppm
Sep 28 14:40:53 dist-test-slave-0n27 krb5kdc[3527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:53.174330 3645 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:40:53.175051 3645 webserver.cc:460] Webserver started at http://127.1.201.1:35535/ using document root <none> and password file <none>
I20260928 14:40:53.175565 3645 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:53.175660 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:53.175899 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:53.177690 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "bf792a1e6849426bb8a219bc451df944"
format_stamp: "Formatted at 2026-09-28 14:40:53 on dist-test-slave-0n27"
server_key: "6aaa3ad481b3c6a0d71bb2bfce750a09"
server_key_iv: "444775ed6ba441d5a735b81c37a1659f"
server_key_version: "encryptionkey@0"
I20260928 14:40:53.178241 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "bf792a1e6849426bb8a219bc451df944"
format_stamp: "Formatted at 2026-09-28 14:40:53 on dist-test-slave-0n27"
server_key: "6aaa3ad481b3c6a0d71bb2bfce750a09"
server_key_iv: "444775ed6ba441d5a735b81c37a1659f"
server_key_version: "encryptionkey@0"
I20260928 14:40:53.181428 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:40:53.183848 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:53.184933 3645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:53.185060 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "bf792a1e6849426bb8a219bc451df944"
format_stamp: "Formatted at 2026-09-28 14:40:53 on dist-test-slave-0n27"
server_key: "6aaa3ad481b3c6a0d71bb2bfce750a09"
server_key_iv: "444775ed6ba441d5a735b81c37a1659f"
server_key_version: "encryptionkey@0"
I20260928 14:40:53.185166 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:53.214293 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:53.217465 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:53.217671 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:53.218959 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:53.219026 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:53.219094 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:53.219135 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:40:53 dist-test-slave-0n27 krb5kdc[3527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:53.229182 3645 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:36919
I20260928 14:40:53.229209 3777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:36919 every 8 connection(s)
I20260928 14:40:53.229995 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:40:53.231281 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.219724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:36419 (local address 127.1.201.62:39223)
0928 14:40:53.219860 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.219867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.220693 (+ 826us) server_negotiation.cc:407] Connection header received
0928 14:40:53.221718 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.221722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.221802 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.221891 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:53.223053 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.223546 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.224621 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.224852 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.224956 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.227864 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:53.227867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:53.227870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:53.227893 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:53.229408 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.229883 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.229886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.229888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.229966 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.230292 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.230296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.230298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.230512 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:53.230597 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.231025 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.231150 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:40:53.232030 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.220091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39223 (local address 127.1.201.1:36419)
0928 14:40:53.220534 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.220577 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.221496 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.222089 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.222097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.222437 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:53.222880 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.222894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.223665 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.223668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.224505 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.224514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.224997 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.225000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.225164 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.225726 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:53.225749 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:53.227720 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:53.229516 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.229521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.229533 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.229762 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.230050 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.230052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.230054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.230194 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.230645 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:53.230652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:53.230846 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.230977 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.231242 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":135,"threads_started":1}
I20260928 14:40:53.233178 3778 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39223
I20260928 14:40:53.233304 3778 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:53.233326 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 3645
I20260928 14:40:53.233448 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:40:53.233738 1828 external_mini_cluster.cc:1428] Setting key 408010feab99ec8afd319895e45f2023
I20260928 14:40:53.233760 3778 heartbeater.cc:511] Master 127.1.201.62:39223 requested a full tablet report, sending...
I20260928 14:40:53.235131 3573 ts_manager.cc:194] Registered new tserver with Master: bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919)
I20260928 14:40:53.236630 3573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:36419
I20260928 14:40:53.242188 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.235292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:60105 (local address 127.1.201.62:39223)
0928 14:40:53.235383 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.235387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.235505 (+ 118us) server_negotiation.cc:407] Connection header received
0928 14:40:53.235612 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.235614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.235666 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.235775 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:53.236164 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.236643 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.237630 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.237940 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.238046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.239171 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:53.239173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:53.239175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:53.239193 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:53.240614 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.241111 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.241114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.241115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.241187 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.241475 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.241477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.241478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.241614 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:53.241684 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.242009 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.242089 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:40:53.242333 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.235206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39223 (local address 127.1.201.1:60105)
0928 14:40:53.235420 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.235434 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.235533 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.235790 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.235793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.235977 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:53.236059 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.236064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.236769 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.236772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.237521 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.237534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.238078 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.238081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.238195 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.238635 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:53.238652 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:53.239049 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:53.240729 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.240734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.240739 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.241017 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.241279 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.241284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.241288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.241388 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.241722 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:53.241727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:53.241864 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.241977 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.242111 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:40:53.258105 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39223
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:53.359324 3787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:53.359549 3787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:53.359622 3787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:53.363416 3787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:53.363487 3787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:53.363565 3787 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:40:53.363605 3787 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:40:53.363660 3787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:40:53.368744 3787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.201.62:39223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:53.369620 3787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:53.370517 3787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:53.378756 3795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:53.378844 3787 server_base.cc:1034] running on GCE node
W20260928 14:40:53.378756 3793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:53.378790 3792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:53.379276 3787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:53.379770 3787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:53.380944 3787 hybrid_clock.cc:648] HybridClock initialized: now 1790606453380933 us; error 20 us; skew 500 ppm
Sep 28 14:40:53 dist-test-slave-0n27 krb5kdc[3527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:53.383273 3787 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:40:53.383989 3787 webserver.cc:460] Webserver started at http://127.1.201.2:41267/ using document root <none> and password file <none>
I20260928 14:40:53.384428 3787 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:53.384477 3787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:53.384609 3787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:53.386415 3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "f680208cf53c4dc7a8132fba43b1a877"
format_stamp: "Formatted at 2026-09-28 14:40:53 on dist-test-slave-0n27"
server_key: "bbbacc91cae4b3087ca6e6a13d10c899"
server_key_iv: "5b8a9763a395a7df6d1811af1ee9788e"
server_key_version: "encryptionkey@0"
I20260928 14:40:53.386940 3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "f680208cf53c4dc7a8132fba43b1a877"
format_stamp: "Formatted at 2026-09-28 14:40:53 on dist-test-slave-0n27"
server_key: "bbbacc91cae4b3087ca6e6a13d10c899"
server_key_iv: "5b8a9763a395a7df6d1811af1ee9788e"
server_key_version: "encryptionkey@0"
I20260928 14:40:53.390118 3787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:40:53.392129 3802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:53.393187 3787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:53.393291 3787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "f680208cf53c4dc7a8132fba43b1a877"
format_stamp: "Formatted at 2026-09-28 14:40:53 on dist-test-slave-0n27"
server_key: "bbbacc91cae4b3087ca6e6a13d10c899"
server_key_iv: "5b8a9763a395a7df6d1811af1ee9788e"
server_key_version: "encryptionkey@0"
I20260928 14:40:53.393376 3787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:53.408298 3787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:53.410625 3787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:53.410768 3787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:53.411811 3787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:53.411854 3787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:53.411885 3787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:53.411937 3787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:40:53 dist-test-slave-0n27 krb5kdc[3527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:53.421604 3787 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:36279
I20260928 14:40:53.421679 3918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:36279 every 8 connection(s)
I20260928 14:40:53.422559 3787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:40:53.422833 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 3787
I20260928 14:40:53.422951 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:40:53.423233 1828 external_mini_cluster.cc:1428] Setting key 9190e6bbe0ce9922568ccc8b173ae2b3
I20260928 14:40:53.425022 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.412635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:36253 (local address 127.1.201.62:39223)
0928 14:40:53.412766 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.412769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.413682 (+ 913us) server_negotiation.cc:407] Connection header received
0928 14:40:53.414749 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.414755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.414825 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.414926 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:53.416378 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.417296 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.418588 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.418968 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.419070 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.421488 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:53.421491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:53.421494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:53.421519 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:53.422968 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.423575 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.423579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.423582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.423673 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.424015 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.424017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.424018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.424182 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:53.424294 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.424768 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.424877 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:40:53.425665 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.413086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39223 (local address 127.1.201.2:36253)
0928 14:40:53.413560 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.413595 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.414489 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.415101 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.415111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.415534 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:53.416211 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.416233 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.417428 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.417434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.418468 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.418477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.419106 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.419110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.419271 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.419792 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:53.419812 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:53.421321 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:53.423191 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.423195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.423206 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.423444 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.423777 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.423781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.423783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.423923 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.424310 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:53.424316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:53.424556 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.424723 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.424974 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260928 14:40:53.426909 3919 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39223
I20260928 14:40:53.427073 3919 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:53.427588 3919 heartbeater.cc:511] Master 127.1.201.62:39223 requested a full tablet report, sending...
I20260928 14:40:53.428530 3573 ts_manager.cc:194] Registered new tserver with Master: f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2:36279)
I20260928 14:40:53.429190 3573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:36253
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
I20260928 14:40:53.435711 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.429035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:36761 (local address 127.1.201.62:39223)
0928 14:40:53.429153 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.429157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.430013 (+ 856us) server_negotiation.cc:407] Connection header received
0928 14:40:53.430112 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.430115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.430149 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.430232 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:53.430591 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.431032 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.431750 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.431960 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.432059 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.432978 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:53.432981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:53.432984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:53.433004 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:53.434217 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.434642 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.434644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.434645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.434703 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.434937 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.434941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.434942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.435162 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:53.435253 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.435475 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.435559 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:40:53.435720 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.428738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39223 (local address 127.1.201.2:36761)
0928 14:40:53.429939 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.429949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.430045 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.430245 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.430248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.430416 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:53.430492 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.430496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.431153 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.431156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.431646 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.431653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.432062 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.432065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.432158 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.432546 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:53.432556 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:53.432865 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:53.434329 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.434333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.434335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.434553 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.434781 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.434785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.434787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.434861 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.435246 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:53.435248 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:53.435323 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.435419 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.435531 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:40:53.448895 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39223
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:53.549531 3929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:53.549724 3929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:53.549762 3929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:53.553449 3929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:53.553490 3929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:53.553542 3929 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:40:53.553555 3929 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:40:53.553584 3929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:40:53.558364 3929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.201.62:39223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:53.559345 3929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:53.560446 3929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:53.570999 3937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:53.571008 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:53.571076 3929 server_base.cc:1034] running on GCE node
W20260928 14:40:53.571008 3935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:53.571720 3929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:53.572274 3929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:53.573452 3929 hybrid_clock.cc:648] HybridClock initialized: now 1790606453573440 us; error 23 us; skew 500 ppm
Sep 28 14:40:53 dist-test-slave-0n27 krb5kdc[3527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:53.575971 3929 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:40:53.576802 3929 webserver.cc:460] Webserver started at http://127.1.201.3:40989/ using document root <none> and password file <none>
I20260928 14:40:53.577287 3929 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:53.577339 3929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:53.577482 3929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:53.579352 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "76b12070c7de4dcfa6e6944c7df8f965"
format_stamp: "Formatted at 2026-09-28 14:40:53 on dist-test-slave-0n27"
server_key: "4570b048906e099849293de59661ec55"
server_key_iv: "7d75d8bf5aa980d08b99896880db52a8"
server_key_version: "encryptionkey@0"
I20260928 14:40:53.579811 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "76b12070c7de4dcfa6e6944c7df8f965"
format_stamp: "Formatted at 2026-09-28 14:40:53 on dist-test-slave-0n27"
server_key: "4570b048906e099849293de59661ec55"
server_key_iv: "7d75d8bf5aa980d08b99896880db52a8"
server_key_version: "encryptionkey@0"
I20260928 14:40:53.582937 3929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 14:40:53.585201 3944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:53.586421 3929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:53.586531 3929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "76b12070c7de4dcfa6e6944c7df8f965"
format_stamp: "Formatted at 2026-09-28 14:40:53 on dist-test-slave-0n27"
server_key: "4570b048906e099849293de59661ec55"
server_key_iv: "7d75d8bf5aa980d08b99896880db52a8"
server_key_version: "encryptionkey@0"
I20260928 14:40:53.586649 3929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:53.603683 3929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:53.606149 3929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:53.606335 3929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:53.607530 3929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:53.607596 3929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:53.607653 3929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:53.607702 3929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:40:53 dist-test-slave-0n27 krb5kdc[3527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:53.617826 3929 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:41337
I20260928 14:40:53.617883 4060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:41337 every 8 connection(s)
I20260928 14:40:53.618592 3929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:40:53.621369 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.608301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:54677 (local address 127.1.201.62:39223)
0928 14:40:53.608444 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.608449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.609237 (+ 788us) server_negotiation.cc:407] Connection header received
0928 14:40:53.610226 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.610231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.610289 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.610350 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:53.612051 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.612946 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.614790 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.615102 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.615187 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.618068 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:53.618072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:53.618074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:53.618098 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:53.619694 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.620218 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.620221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.620222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.620289 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.620563 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.620565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.620566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.620714 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:53.620782 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.621170 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.621263 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:40:53.622109 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.608673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39223 (local address 127.1.201.3:54677)
0928 14:40:53.609078 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.609131 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.609990 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.610716 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.610725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.611207 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:53.611900 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.611925 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.613091 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.613097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.614642 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.614656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.615268 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.615274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.615482 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.616103 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:53.616127 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:53.617919 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:53.619799 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.619820 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.619835 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.620116 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.620362 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.620364 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.620365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.620483 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.620797 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:53.620801 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:53.620983 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.621125 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.621332 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":119,"threads_started":1}
I20260928 14:40:53.623207 4061 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39223
I20260928 14:40:53.623353 4061 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:53.623819 4061 heartbeater.cc:511] Master 127.1.201.62:39223 requested a full tablet report, sending...
I20260928 14:40:53.624046 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 3929
I20260928 14:40:53.624135 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:40:53.624389 1828 external_mini_cluster.cc:1428] Setting key 6f5a9a62ba4423b2630317cfbc4bc67f
I20260928 14:40:53.624818 3573 ts_manager.cc:194] Registered new tserver with Master: 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337)
I20260928 14:40:53.625432 3573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:54677
I20260928 14:40:53.626447 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:40:53.632467 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.625135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:50581 (local address 127.1.201.62:39223)
0928 14:40:53.625214 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.625218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.625548 (+ 330us) server_negotiation.cc:407] Connection header received
0928 14:40:53.625718 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.625721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.625759 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.625822 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:53.626427 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.627142 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.628360 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.628550 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.628666 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.629647 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:53.629651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:53.629653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:53.629675 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:53.631044 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.631441 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.631445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.631446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.631521 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.631776 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.631780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.631782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.631929 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:53.632024 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.632259 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.632350 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:40:53.632477 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.625038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39223 (local address 127.1.201.3:50581)
0928 14:40:53.625483 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.625495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.625607 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.625978 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.625981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.626170 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:53.626304 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.626312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.627316 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.627322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.628246 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.628260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.628670 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.628674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.628767 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.629245 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:53.629256 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:53.629512 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:53.631169 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.631172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.631174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.631349 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.631605 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.631609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.631614 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.631693 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.632017 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:53.632019 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:53.632110 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.632208 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.632326 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":94,"threads_started":1}
Sep 28 14:40:53 dist-test-slave-0n27 krb5kdc[3527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:40:53 dist-test-slave-0n27 krb5kdc[3527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:53.650717 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.643365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.1.201.62:39223)
0928 14:40:53.643472 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.643475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.643486 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:40:53.643513 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.643515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.643544 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.643608 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:53.644209 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.644662 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.645586 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.645844 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.645959 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.648143 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:53.648146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:53.648148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:53.648168 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:53.649467 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.649844 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.649846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.649847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.649913 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.650137 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.650139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.650140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.650273 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:53.650364 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.650525 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.650609 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:40:53.653235 3572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:40:53.656068 3572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:40:53.669432 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.663442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:41337 (local address 127.0.0.1:34288)
0928 14:40:53.663975 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.664019 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.664125 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.665145 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.665150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.665184 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:53.665343 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.665349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.666466 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.666472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.667966 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.667976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.669028 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.669034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.669140 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.669159 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.669233 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.669284 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":157,"spinlock_wait_cycles":9088,"thread_start_us":80,"threads_started":1}
I20260928 14:40:53.669494 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.664541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.1.201.1:36919)
0928 14:40:53.664930 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.664940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.665209 (+ 269us) server_negotiation.cc:407] Connection header received
0928 14:40:53.665304 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.665311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.665480 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.665618 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:53.665883 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.666971 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.668160 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.668675 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.668795 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.668961 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.669249 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.669342 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":96,"threads_started":1}
I20260928 14:40:53.669608 4080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.663548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.1.201.3:41337)
0928 14:40:53.664683 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.664695 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.664715 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:40:53.664776 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.664783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.664957 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.665145 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:53.665449 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.666355 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.668190 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.668888 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.669019 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.669158 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.669383 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.669478 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1008,"thread_start_us":96,"threads_started":1}
I20260928 14:40:53.669312 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.664101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:36919 (local address 127.0.0.1:53082)
0928 14:40:53.665014 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.665026 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.665107 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.665609 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.665612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.665631 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:53.665719 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.665725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.667100 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.667106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.668018 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.668050 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.668795 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.668798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.668905 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.668953 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.669050 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.669169 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":29,"thread_start_us":124,"threads_started":1}
I20260928 14:40:53.669860 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.663694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:36279 (local address 127.0.0.1:33918)
0928 14:40:53.664336 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.664350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.664432 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.665131 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.665151 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.665195 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:53.665358 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.665366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.666779 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.666782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.667966 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.667976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.669230 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.669234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.669316 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.669337 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.669403 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.669462 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":109,"spinlock_wait_cycles":31360,"thread_start_us":89,"threads_started":1}
I20260928 14:40:53.670665 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.664167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.1.201.2:36279)
0928 14:40:53.664494 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.664506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.664528 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:40:53.664600 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.664605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.664804 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.664965 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:53.665481 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.666665 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.668090 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.668878 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.668998 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.669159 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.670193 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.670274 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
I20260928 14:40:53.672246 3710 tablet_service.cc:1467] Processing CreateTablet for tablet fe8a3db2ed0c4fb68291a7e041b31200 (DEFAULT_TABLE table=test-table [id=6a4e454d44df460eb6a1dcf8f5a84a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:53.672246 3994 tablet_service.cc:1467] Processing CreateTablet for tablet fe8a3db2ed0c4fb68291a7e041b31200 (DEFAULT_TABLE table=test-table [id=6a4e454d44df460eb6a1dcf8f5a84a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:53.672616 3853 tablet_service.cc:1467] Processing CreateTablet for tablet fe8a3db2ed0c4fb68291a7e041b31200 (DEFAULT_TABLE table=test-table [id=6a4e454d44df460eb6a1dcf8f5a84a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:53.673214 3710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8a3db2ed0c4fb68291a7e041b31200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:53.673219 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8a3db2ed0c4fb68291a7e041b31200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:53.673491 3853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8a3db2ed0c4fb68291a7e041b31200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:53.679363 4085 tablet_bootstrap.cc:492] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: Bootstrap starting.
I20260928 14:40:53.680529 4086 tablet_bootstrap.cc:492] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: Bootstrap starting.
I20260928 14:40:53.681682 4085 tablet_bootstrap.cc:654] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:53.682420 4086 tablet_bootstrap.cc:654] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:53.682495 4085 log.cc:824] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:53.683099 4087 tablet_bootstrap.cc:492] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: Bootstrap starting.
I20260928 14:40:53.683153 4086 log.cc:824] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:53.684928 4087 tablet_bootstrap.cc:654] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:53.685019 4086 tablet_bootstrap.cc:492] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: No bootstrap required, opened a new log
I20260928 14:40:53.685210 4086 ts_tablet_manager.cc:1397] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:40:53.685667 4087 log.cc:824] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:53.687630 4087 tablet_bootstrap.cc:492] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: No bootstrap required, opened a new log
I20260928 14:40:53.687794 4087 ts_tablet_manager.cc:1397] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:40:53.687628 4085 tablet_bootstrap.cc:492] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: No bootstrap required, opened a new log
I20260928 14:40:53.688356 4086 raft_consensus.cc:348] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } }
I20260928 14:40:53.688561 4086 raft_consensus.cc:374] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:53.688601 4085 ts_tablet_manager.cc:1397] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 14:40:53.688617 4086 raft_consensus.cc:729] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf792a1e6849426bb8a219bc451df944, State: Initialized, Role: FOLLOWER
I20260928 14:40:53.689024 4086 consensus_queue.cc:260] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } }
I20260928 14:40:53.689567 3778 heartbeater.cc:503] Master 127.1.201.62:39223 was elected leader, sending a full tablet report...
I20260928 14:40:53.689894 4086 ts_tablet_manager.cc:1428] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:40:53.691541 4087 raft_consensus.cc:348] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } }
I20260928 14:40:53.691789 4087 raft_consensus.cc:374] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:53.691843 4087 raft_consensus.cc:729] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b12070c7de4dcfa6e6944c7df8f965, State: Initialized, Role: FOLLOWER
I20260928 14:40:53.692214 4087 consensus_queue.cc:260] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } }
I20260928 14:40:53.692225 4085 raft_consensus.cc:348] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } }
I20260928 14:40:53.692440 4085 raft_consensus.cc:374] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:53.692513 4085 raft_consensus.cc:729] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f680208cf53c4dc7a8132fba43b1a877, State: Initialized, Role: FOLLOWER
I20260928 14:40:53.692878 4085 consensus_queue.cc:260] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } }
I20260928 14:40:53.693038 4061 heartbeater.cc:503] Master 127.1.201.62:39223 was elected leader, sending a full tablet report...
I20260928 14:40:53.693130 4087 ts_tablet_manager.cc:1428] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:40:53.693710 4085 ts_tablet_manager.cc:1428] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:40:53.694065 3919 heartbeater.cc:503] Master 127.1.201.62:39223 was elected leader, sending a full tablet report...
W20260928 14:40:53.731000 3779 tablet.cc:2369] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:40:53.776297 4093 raft_consensus.cc:482] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:40:53.776494 4093 raft_consensus.cc:504] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } }
I20260928 14:40:53.777544 4093 leader_election.cc:288] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337), bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919)
I20260928 14:40:53.781152 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.777822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:41337 (local address 127.1.201.2:54225)
0928 14:40:53.777947 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.777961 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.778048 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.778319 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.778322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.778341 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:53.778482 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.778491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.779285 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.779287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.780229 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.780238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.780852 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.780854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.780925 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.780943 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.780997 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.781048 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 14:40:53.781212 4080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.777928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:54225 (local address 127.1.201.3:41337)
0928 14:40:53.778065 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.778070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.778091 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:40:53.778149 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.778154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.778207 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.778296 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:53.778597 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.779155 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.780342 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.780720 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.780836 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.780894 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.781031 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.781093 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:40:53.781719 4014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8a3db2ed0c4fb68291a7e041b31200" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b12070c7de4dcfa6e6944c7df8f965" is_pre_election: true
I20260928 14:40:53.781813 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.777822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:36919 (local address 127.1.201.2:42685)
0928 14:40:53.778345 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.778357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.778443 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.778710 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.778712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.778726 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:53.779136 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.779141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.779808 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.779811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.780689 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.780697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.781480 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.781483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.781575 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.781590 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.781643 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.781684 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":160,"thread_start_us":147,"threads_started":1}
I20260928 14:40:53.781859 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.777962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:42685 (local address 127.1.201.1:36919)
0928 14:40:53.778070 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.778075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.778444 (+ 369us) server_negotiation.cc:407] Connection header received
0928 14:40:53.778530 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.778560 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.778614 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.778692 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:53.779249 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.779712 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.780799 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.781360 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.781466 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.781519 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.781677 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.781745 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:40:53.782006 4014 raft_consensus.cc:2454] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 0.
I20260928 14:40:53.782414 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8a3db2ed0c4fb68291a7e041b31200" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf792a1e6849426bb8a219bc451df944" is_pre_election: true
I20260928 14:40:53.782469 3804 leader_election.cc:302] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b12070c7de4dcfa6e6944c7df8f965, f680208cf53c4dc7a8132fba43b1a877; no voters:
I20260928 14:40:53.782660 3731 raft_consensus.cc:2454] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 0.
I20260928 14:40:53.782680 4093 raft_consensus.cc:2781] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:40:53.782742 4093 raft_consensus.cc:482] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:40:53.782823 4093 raft_consensus.cc:3037] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:53.784026 4093 raft_consensus.cc:504] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } }
I20260928 14:40:53.784402 4093 leader_election.cc:288] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 election: Requested vote from peers 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337), bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919)
I20260928 14:40:53.784718 4014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8a3db2ed0c4fb68291a7e041b31200" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b12070c7de4dcfa6e6944c7df8f965"
I20260928 14:40:53.784734 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8a3db2ed0c4fb68291a7e041b31200" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf792a1e6849426bb8a219bc451df944"
I20260928 14:40:53.784801 4014 raft_consensus.cc:3037] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:53.784873 3731 raft_consensus.cc:3037] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:53.785933 4014 raft_consensus.cc:2454] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 1.
I20260928 14:40:53.785977 3731 raft_consensus.cc:2454] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 1.
I20260928 14:40:53.786244 3804 leader_election.cc:302] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b12070c7de4dcfa6e6944c7df8f965, f680208cf53c4dc7a8132fba43b1a877; no voters:
I20260928 14:40:53.786425 4093 raft_consensus.cc:2781] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:53.786618 4093 raft_consensus.cc:686] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 1 LEADER]: Becoming Leader. State: Replica: f680208cf53c4dc7a8132fba43b1a877, State: Running, Role: LEADER
I20260928 14:40:53.786950 4093 consensus_queue.cc:237] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } }
I20260928 14:40:53.789817 3573 catalog_manager.cc:5495] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 reported cstate change: term changed from 0 to 1, leader changed from <none> to f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2). New cstate: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:40:53 dist-test-slave-0n27 krb5kdc[3527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:53.838305 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.830984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39223 (local address 127.0.0.1:49060)
0928 14:40:53.831365 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.831375 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:53.831489 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:53.831737 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:53.831739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:53.832096 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:53.832166 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.832172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.832860 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.832865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.833398 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:53.833405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.833857 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.833859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:53.833919 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.834378 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:53.834389 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:53.835690 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:53.837062 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.837069 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.837073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.837270 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.837480 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:53.837482 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:53.837483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:53.837550 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:53.837881 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:53.837885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:53.837968 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 14:40:53.838049 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:40:53.838170 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":93,"threads_started":1}
I20260928 14:40:53.838327 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.831067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.1.201.62:39223)
0928 14:40:53.831306 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:53.831311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:53.831441 (+ 130us) server_negotiation.cc:407] Connection header received
0928 14:40:53.831575 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:53.831579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:53.831621 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:53.831720 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:53.832278 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.832728 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.833543 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:53.833743 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:53.833831 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:53.835808 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:53.835811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:53.835812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:53.835832 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:53.836963 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.837345 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.837347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.837348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.837409 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:53.837616 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:53.837618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:53.837618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:53.837755 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:53.837878 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:53.838099 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 14:40:53.838217 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":161}
I20260928 14:40:53.841287 3573 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:49060:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:40:53.842031 3573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:40:53.846287 3710 tablet_service.cc:1467] Processing CreateTablet for tablet 3570088eb06d4670a177b76cece75af0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba39d9f03364481aae8a7f0c5792ebb4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:40:53.846421 3853 tablet_service.cc:1467] Processing CreateTablet for tablet 3570088eb06d4670a177b76cece75af0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba39d9f03364481aae8a7f0c5792ebb4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:40:53.846997 3710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3570088eb06d4670a177b76cece75af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:53.847043 3853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3570088eb06d4670a177b76cece75af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:53.846724 3994 tablet_service.cc:1467] Processing CreateTablet for tablet 3570088eb06d4670a177b76cece75af0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba39d9f03364481aae8a7f0c5792ebb4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:40:53.847267 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3570088eb06d4670a177b76cece75af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:53.849644 4085 tablet_bootstrap.cc:492] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: Bootstrap starting.
I20260928 14:40:53.849880 4086 tablet_bootstrap.cc:492] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: Bootstrap starting.
I20260928 14:40:53.849887 4087 tablet_bootstrap.cc:492] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: Bootstrap starting.
I20260928 14:40:53.850606 4085 tablet_bootstrap.cc:654] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:53.850623 4087 tablet_bootstrap.cc:654] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:53.850714 4086 tablet_bootstrap.cc:654] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:53.851877 4087 tablet_bootstrap.cc:492] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: No bootstrap required, opened a new log
I20260928 14:40:53.851971 4087 ts_tablet_manager.cc:1397] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:40:53.852300 4086 tablet_bootstrap.cc:492] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: No bootstrap required, opened a new log
I20260928 14:40:53.852382 4086 ts_tablet_manager.cc:1397] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:40:53.852432 4087 raft_consensus.cc:348] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:53.852533 4087 raft_consensus.cc:374] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:53.852550 4085 tablet_bootstrap.cc:492] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: No bootstrap required, opened a new log
I20260928 14:40:53.852571 4087 raft_consensus.cc:729] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b12070c7de4dcfa6e6944c7df8f965, State: Initialized, Role: FOLLOWER
I20260928 14:40:53.852614 4085 ts_tablet_manager.cc:1397] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:40:53.852705 4087 consensus_queue.cc:260] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:53.852839 4086 raft_consensus.cc:348] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:53.852941 4086 raft_consensus.cc:374] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:53.852973 4086 raft_consensus.cc:729] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf792a1e6849426bb8a219bc451df944, State: Initialized, Role: FOLLOWER
I20260928 14:40:53.853001 4087 ts_tablet_manager.cc:1428] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:53.852967 4092 tablet_replica.cc:439] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } }
I20260928 14:40:53.853096 4092 tablet_replica.cc:442] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:53.853036 4085 raft_consensus.cc:348] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:53.853098 4086 consensus_queue.cc:260] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:53.853148 4085 raft_consensus.cc:374] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:53.853179 4085 raft_consensus.cc:729] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f680208cf53c4dc7a8132fba43b1a877, State: Initialized, Role: FOLLOWER
I20260928 14:40:53.853299 4085 consensus_queue.cc:260] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:53.853340 4091 tablet_replica.cc:439] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } }
I20260928 14:40:53.853482 4091 tablet_replica.cc:442] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:53.853529 4085 ts_tablet_manager.cc:1428] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:53.853363 4086 ts_tablet_manager.cc:1428] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:53.854104 4093 tablet_replica.cc:439] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } }
I20260928 14:40:53.854240 4093 tablet_replica.cc:442] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 14:40:53.869593 4062 tablet.cc:2369] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:40:53.923607 3920 tablet.cc:2369] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:40:53.962211 4093 raft_consensus.cc:482] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:40:53.962320 4093 raft_consensus.cc:504] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:53.962738 4093 leader_election.cc:288] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919), 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337)
I20260928 14:40:53.963117 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3570088eb06d4670a177b76cece75af0" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf792a1e6849426bb8a219bc451df944" is_pre_election: true
I20260928 14:40:53.963204 4014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3570088eb06d4670a177b76cece75af0" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b12070c7de4dcfa6e6944c7df8f965" is_pre_election: true
I20260928 14:40:53.963276 3731 raft_consensus.cc:2454] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 0.
I20260928 14:40:53.963394 4014 raft_consensus.cc:2454] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 0.
I20260928 14:40:53.963582 3806 leader_election.cc:302] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf792a1e6849426bb8a219bc451df944, f680208cf53c4dc7a8132fba43b1a877; no voters:
I20260928 14:40:53.963760 4093 raft_consensus.cc:2781] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:40:53.963828 4093 raft_consensus.cc:482] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:40:53.963869 4093 raft_consensus.cc:3037] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:53.965046 4093 raft_consensus.cc:504] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:53.965363 4093 leader_election.cc:288] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 election: Requested vote from peers bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919), 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337)
I20260928 14:40:53.965682 4014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3570088eb06d4670a177b76cece75af0" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b12070c7de4dcfa6e6944c7df8f965"
I20260928 14:40:53.965728 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3570088eb06d4670a177b76cece75af0" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf792a1e6849426bb8a219bc451df944"
I20260928 14:40:53.965880 3731 raft_consensus.cc:3037] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:53.965881 4014 raft_consensus.cc:3037] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:53.966879 4014 raft_consensus.cc:2454] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 1.
I20260928 14:40:53.966948 3731 raft_consensus.cc:2454] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 1.
I20260928 14:40:53.967166 3804 leader_election.cc:302] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b12070c7de4dcfa6e6944c7df8f965, f680208cf53c4dc7a8132fba43b1a877; no voters:
I20260928 14:40:53.967365 4093 raft_consensus.cc:2781] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:53.967432 4093 raft_consensus.cc:686] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [term 1 LEADER]: Becoming Leader. State: Replica: f680208cf53c4dc7a8132fba43b1a877, State: Running, Role: LEADER
I20260928 14:40:53.967543 4093 consensus_queue.cc:237] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:53.968081 4093 tablet_replica.cc:439] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: TxnStatusTablet state changed. Reason: New leader f680208cf53c4dc7a8132fba43b1a877. Latest consensus state: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } }
I20260928 14:40:53.968165 4093 tablet_replica.cc:442] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:40:53.968403 4107 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:40:53.968581 4107 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:40:53.969106 3571 catalog_manager.cc:5495] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 reported cstate change: term changed from 0 to 1, leader changed from <none> to f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2). New cstate: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } health_report { overall_health: UNKNOWN } } }
I20260928 14:40:54.003731 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.999770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:36279 (local address 127.0.0.1:33934)
0928 14:40:53.999928 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:53.999945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:54.000046 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:54.000416 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:54.000418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:54.000424 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:40:54.000509 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:54.000513 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:54.001516 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:54.001518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:54.002176 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:54.002187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:54.002742 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:54.002744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:54.002809 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:54.002850 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:40:54.003514 (+ 664us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:40:54.003525 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 14:40:54.003564 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 14:40:54.003626 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 14:40:54.003759 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:53.999904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.1.201.2:36279)
0928 14:40:54.000011 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:54.000015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:54.000040 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:40:54.000168 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:54.000173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:54.000261 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:54.000404 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:54.000607 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:54.001419 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:54.002288 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:54.002639 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:54.002738 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:54.002978 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:54.003427 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:54.003531 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:54.003591 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 14:40:54.003642 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:40:54.008225 4014 raft_consensus.cc:1261] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Refusing update from remote peer f680208cf53c4dc7a8132fba43b1a877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:54.008361 3731 raft_consensus.cc:1261] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Refusing update from remote peer f680208cf53c4dc7a8132fba43b1a877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:54.008934 4095 consensus_queue.cc:1035] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:54.009131 4093 consensus_queue.cc:1035] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:54.012249 4091 tablet_replica.cc:439] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: TxnStatusTablet state changed. Reason: New leader f680208cf53c4dc7a8132fba43b1a877. Latest consensus state: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } }
I20260928 14:40:54.012365 4091 tablet_replica.cc:442] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:54.012630 4092 tablet_replica.cc:439] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: TxnStatusTablet state changed. Reason: New leader f680208cf53c4dc7a8132fba43b1a877. Latest consensus state: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } }
I20260928 14:40:54.012740 4092 tablet_replica.cc:442] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:54.015264 4095 tablet_replica.cc:439] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } }
I20260928 14:40:54.015360 4095 tablet_replica.cc:442] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:40:54.015547 4110 mvcc.cc:205] Tried to move back new op lower bound from 7334324035611701248 to 7334324035452121088. Current Snapshot: MvccSnapshot[applied={T|T < 7334324035611701248}]
I20260928 14:40:54.015929 4093 tablet_replica.cc:439] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } }
I20260928 14:40:54.016016 4093 tablet_replica.cc:442] T 3570088eb06d4670a177b76cece75af0 P f680208cf53c4dc7a8132fba43b1a877: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:40:54.016918 4091 tablet_replica.cc:439] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } }
I20260928 14:40:54.017014 4091 tablet_replica.cc:442] T 3570088eb06d4670a177b76cece75af0 P bf792a1e6849426bb8a219bc451df944: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:54.017875 4111 mvcc.cc:205] Tried to move back new op lower bound from 7334324035611701248 to 7334324035452121088. Current Snapshot: MvccSnapshot[applied={T|T < 7334324035611701248}]
I20260928 14:40:54.017930 4112 mvcc.cc:205] Tried to move back new op lower bound from 7334324035611701248 to 7334324035452121088. Current Snapshot: MvccSnapshot[applied={T|T < 7334324035611701248}]
I20260928 14:40:54.017987 4092 tablet_replica.cc:439] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } } }
I20260928 14:40:54.018064 4092 tablet_replica.cc:442] T 3570088eb06d4670a177b76cece75af0 P 76b12070c7de4dcfa6e6944c7df8f965: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:40:54.025149 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:54.022403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.1.201.2:36279)
0928 14:40:54.022499 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:54.022504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:54.022517 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:40:54.022552 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:54.022554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:54.022588 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:54.022676 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:54.022927 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:54.023406 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:54.024213 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:54.024410 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:54.024529 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:54.024773 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:54.024839 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:54.024912 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:54.024999 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 14:40:54.025040 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:40:54.034699 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:54.031301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:55727 (local address 127.1.201.2:36279)
0928 14:40:54.031403 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:54.031407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:54.031705 (+ 298us) server_negotiation.cc:407] Connection header received
0928 14:40:54.031829 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:54.031831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:54.031864 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:54.031919 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:54.032167 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:54.032554 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:54.033688 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:54.033936 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:54.034005 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:54.034305 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:54.034381 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:54.034437 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:54.034557 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 14:40:54.034590 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:40:54.034756 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:54.031231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:36279 (local address 127.1.201.2:55727)
0928 14:40:54.031615 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:54.031632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:54.031745 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:54.031958 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:54.031961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:54.031969 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:40:54.032069 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:54.032074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:54.032651 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:54.032654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:54.033581 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:54.033591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:54.034035 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:54.034039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:54.034124 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:54.034165 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:40:54.034467 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:40:54.034473 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:40:54.034540 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:40:54.034611 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":99,"threads_started":1}
I20260928 14:40:54.039992 3731 raft_consensus.cc:1261] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Refusing update from remote peer f680208cf53c4dc7a8132fba43b1a877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:54.040123 4014 raft_consensus.cc:1261] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Refusing update from remote peer f680208cf53c4dc7a8132fba43b1a877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:54.040472 4122 consensus_queue.cc:1035] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:54.040628 4095 consensus_queue.cc:1035] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:54.079804 3572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:54.079957 3572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:54.081974 3572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4: Sending DeleteTablet for 3 replicas of tablet fe8a3db2ed0c4fb68291a7e041b31200
I20260928 14:40:54.082541 3994 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8a3db2ed0c4fb68291a7e041b31200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:54 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:34288
I20260928 14:40:54.083010 4126 tablet_replica.cc:330] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: stopping tablet replica
I20260928 14:40:54.083077 3710 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8a3db2ed0c4fb68291a7e041b31200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:54 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:53082
I20260928 14:40:54.083217 3572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:40:54.083472 3852 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8a3db2ed0c4fb68291a7e041b31200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:54 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:33918
W20260928 14:40:54.083561 3572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:40:54.083534 4126 raft_consensus.cc:2229] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:54.083864 4127 tablet_replica.cc:330] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: stopping tablet replica
I20260928 14:40:54.083868 4126 raft_consensus.cc:2258] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:54.084113 4127 raft_consensus.cc:2229] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:54.084295 4127 raft_consensus.cc:2258] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:54.085191 4128 tablet_replica.cc:330] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: stopping tablet replica
I20260928 14:40:54.085429 4128 raft_consensus.cc:2229] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:40:54.085665 4126 ts_tablet_manager.cc:1905] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:54.085713 4128 raft_consensus.cc:2258] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:54.087893 4127 ts_tablet_manager.cc:1905] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:54.088940 4126 ts_tablet_manager.cc:1918] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:40:54.089082 4126 log.cc:1196] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/fe8a3db2ed0c4fb68291a7e041b31200
I20260928 14:40:54.089280 3710 tablet_service.cc:1467] Processing CreateTablet for tablet 2149de7157224507bea78337507b471f (DEFAULT_TABLE table=test-table [id=014fafa350ce4528aff510dbd78b341e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:54.089550 3710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2149de7157224507bea78337507b471f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:54.089553 4126 ts_tablet_manager.cc:1939] T fe8a3db2ed0c4fb68291a7e041b31200 P 76b12070c7de4dcfa6e6944c7df8f965: Deleting consensus metadata
I20260928 14:40:54.090948 4127 ts_tablet_manager.cc:1918] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:40:54.091044 4127 log.cc:1196] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/fe8a3db2ed0c4fb68291a7e041b31200
I20260928 14:40:54.091379 4127 ts_tablet_manager.cc:1939] T fe8a3db2ed0c4fb68291a7e041b31200 P bf792a1e6849426bb8a219bc451df944: Deleting consensus metadata
I20260928 14:40:54.091729 4086 tablet_bootstrap.cc:492] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944: Bootstrap starting.
I20260928 14:40:54.091897 3994 tablet_service.cc:1467] Processing CreateTablet for tablet 2149de7157224507bea78337507b471f (DEFAULT_TABLE table=test-table [id=014fafa350ce4528aff510dbd78b341e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:54.092098 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2149de7157224507bea78337507b471f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:54.092584 4086 tablet_bootstrap.cc:654] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:54.093713 4086 tablet_bootstrap.cc:492] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944: No bootstrap required, opened a new log
I20260928 14:40:54.093880 4086 ts_tablet_manager.cc:1397] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:54.094300 4087 tablet_bootstrap.cc:492] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965: Bootstrap starting.
I20260928 14:40:54.094302 4086 raft_consensus.cc:348] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:54.094417 4086 raft_consensus.cc:374] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:54.094452 4086 raft_consensus.cc:729] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf792a1e6849426bb8a219bc451df944, State: Initialized, Role: FOLLOWER
I20260928 14:40:54.094569 3560 catalog_manager.cc:4841] TS bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919): tablet fe8a3db2ed0c4fb68291a7e041b31200 (table test-table [id=6a4e454d44df460eb6a1dcf8f5a84a82]) successfully deleted
I20260928 14:40:54.094554 4086 consensus_queue.cc:260] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:54.094837 4086 ts_tablet_manager.cc:1428] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:54.094848 3558 catalog_manager.cc:4841] TS 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337): tablet fe8a3db2ed0c4fb68291a7e041b31200 (table test-table [id=6a4e454d44df460eb6a1dcf8f5a84a82]) successfully deleted
I20260928 14:40:54.095129 4087 tablet_bootstrap.cc:654] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:54.095546 3852 tablet_service.cc:1467] Processing CreateTablet for tablet 2149de7157224507bea78337507b471f (DEFAULT_TABLE table=test-table [id=014fafa350ce4528aff510dbd78b341e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:54.095866 3852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2149de7157224507bea78337507b471f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:54.097178 4087 tablet_bootstrap.cc:492] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965: No bootstrap required, opened a new log
I20260928 14:40:54.097282 4087 ts_tablet_manager.cc:1397] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:40:54.098074 4087 raft_consensus.cc:348] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:54.098198 4087 raft_consensus.cc:374] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:54.098242 4087 raft_consensus.cc:729] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b12070c7de4dcfa6e6944c7df8f965, State: Initialized, Role: FOLLOWER
I20260928 14:40:54.098627 4087 consensus_queue.cc:260] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:54.098767 4085 tablet_bootstrap.cc:492] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877: Bootstrap starting.
I20260928 14:40:54.098975 4087 ts_tablet_manager.cc:1428] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:40:54.099588 4085 tablet_bootstrap.cc:654] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:54.100108 4128 ts_tablet_manager.cc:1905] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:54.102593 4128 ts_tablet_manager.cc:1918] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:40:54.102694 4128 log.cc:1196] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/fe8a3db2ed0c4fb68291a7e041b31200
I20260928 14:40:54.102962 4128 ts_tablet_manager.cc:1939] T fe8a3db2ed0c4fb68291a7e041b31200 P f680208cf53c4dc7a8132fba43b1a877: Deleting consensus metadata
I20260928 14:40:54.103806 3559 catalog_manager.cc:4841] TS f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2:36279): tablet fe8a3db2ed0c4fb68291a7e041b31200 (table test-table [id=6a4e454d44df460eb6a1dcf8f5a84a82]) successfully deleted
I20260928 14:40:54.104352 4085 tablet_bootstrap.cc:492] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877: No bootstrap required, opened a new log
I20260928 14:40:54.104435 4085 ts_tablet_manager.cc:1397] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 14:40:54.104828 4085 raft_consensus.cc:348] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:54.104944 4085 raft_consensus.cc:374] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:54.104987 4085 raft_consensus.cc:729] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f680208cf53c4dc7a8132fba43b1a877, State: Initialized, Role: FOLLOWER
I20260928 14:40:54.105101 4085 consensus_queue.cc:260] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:54.105361 4085 ts_tablet_manager.cc:1428] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:54.140331 4122 raft_consensus.cc:482] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:40:54.140429 4122 raft_consensus.cc:504] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:54.140756 4122 leader_election.cc:288] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919), 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337)
I20260928 14:40:54.141063 4013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2149de7157224507bea78337507b471f" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b12070c7de4dcfa6e6944c7df8f965" is_pre_election: true
I20260928 14:40:54.141058 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2149de7157224507bea78337507b471f" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf792a1e6849426bb8a219bc451df944" is_pre_election: true
I20260928 14:40:54.141197 4013 raft_consensus.cc:2454] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 0.
I20260928 14:40:54.141206 3731 raft_consensus.cc:2454] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 0.
I20260928 14:40:54.141463 3806 leader_election.cc:302] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf792a1e6849426bb8a219bc451df944, f680208cf53c4dc7a8132fba43b1a877; no voters:
I20260928 14:40:54.141620 4122 raft_consensus.cc:2781] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:40:54.141681 4122 raft_consensus.cc:482] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:40:54.141724 4122 raft_consensus.cc:3037] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:54.142799 4122 raft_consensus.cc:504] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:54.143115 4122 leader_election.cc:288] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 election: Requested vote from peers bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919), 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337)
I20260928 14:40:54.143412 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2149de7157224507bea78337507b471f" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf792a1e6849426bb8a219bc451df944"
I20260928 14:40:54.143461 4013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2149de7157224507bea78337507b471f" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b12070c7de4dcfa6e6944c7df8f965"
I20260928 14:40:54.143515 3731 raft_consensus.cc:3037] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:54.143561 4013 raft_consensus.cc:3037] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:54.144553 4013 raft_consensus.cc:2454] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 1.
I20260928 14:40:54.144553 3731 raft_consensus.cc:2454] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 1.
I20260928 14:40:54.144816 3804 leader_election.cc:302] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b12070c7de4dcfa6e6944c7df8f965, f680208cf53c4dc7a8132fba43b1a877; no voters:
I20260928 14:40:54.144994 4122 raft_consensus.cc:2781] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:54.145087 4122 raft_consensus.cc:686] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 1 LEADER]: Becoming Leader. State: Replica: f680208cf53c4dc7a8132fba43b1a877, State: Running, Role: LEADER
I20260928 14:40:54.145246 4122 consensus_queue.cc:237] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:54.146705 3571 catalog_manager.cc:5495] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 reported cstate change: term changed from 0 to 1, leader changed from <none> to f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2). New cstate: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } health_report { overall_health: UNKNOWN } } }
I20260928 14:40:54.171525 3731 raft_consensus.cc:1261] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Refusing update from remote peer f680208cf53c4dc7a8132fba43b1a877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:54.171706 4013 raft_consensus.cc:1261] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Refusing update from remote peer f680208cf53c4dc7a8132fba43b1a877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:54.172066 4133 consensus_queue.cc:1035] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:54.172207 4093 consensus_queue.cc:1035] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:40:54.185168 4110 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:40:55.732813 3572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:55.732939 3572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:55.735214 3572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4: Sending DeleteTablet for 3 replicas of tablet 2149de7157224507bea78337507b471f
I20260928 14:40:55.735749 3710 tablet_service.cc:1514] Processing DeleteTablet for tablet 2149de7157224507bea78337507b471f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:55 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:53082
I20260928 14:40:55.735807 3852 tablet_service.cc:1514] Processing DeleteTablet for tablet 2149de7157224507bea78337507b471f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:55 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:33918
I20260928 14:40:55.736029 3994 tablet_service.cc:1514] Processing DeleteTablet for tablet 2149de7157224507bea78337507b471f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:55 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:34288
I20260928 14:40:55.736056 4156 tablet_replica.cc:330] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944: stopping tablet replica
I20260928 14:40:55.736234 4156 raft_consensus.cc:2229] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:55.736330 4158 tablet_replica.cc:330] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965: stopping tablet replica
I20260928 14:40:55.736354 4156 raft_consensus.cc:2258] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:55.736476 4158 raft_consensus.cc:2229] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:55.736588 4158 raft_consensus.cc:2258] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:55.736816 4157 tablet_replica.cc:330] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877: stopping tablet replica
I20260928 14:40:55.736966 4157 raft_consensus.cc:2229] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:40:55.737128 4156 ts_tablet_manager.cc:1905] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:55.737146 4157 raft_consensus.cc:2258] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:55.737303 4158 ts_tablet_manager.cc:1905] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:55.738569 4157 ts_tablet_manager.cc:1905] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:55.739629 4158 ts_tablet_manager.cc:1918] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:40:55.739627 4156 ts_tablet_manager.cc:1918] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:40:55.739705 4158 log.cc:1196] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/2149de7157224507bea78337507b471f
I20260928 14:40:55.739713 4156 log.cc:1196] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/2149de7157224507bea78337507b471f
I20260928 14:40:55.739951 4156 ts_tablet_manager.cc:1939] T 2149de7157224507bea78337507b471f P bf792a1e6849426bb8a219bc451df944: Deleting consensus metadata
I20260928 14:40:55.739951 4158 ts_tablet_manager.cc:1939] T 2149de7157224507bea78337507b471f P 76b12070c7de4dcfa6e6944c7df8f965: Deleting consensus metadata
I20260928 14:40:55.740497 3558 catalog_manager.cc:4841] TS 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337): tablet 2149de7157224507bea78337507b471f (table test-table [id=014fafa350ce4528aff510dbd78b341e]) successfully deleted
I20260928 14:40:55.740558 3560 catalog_manager.cc:4841] TS bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919): tablet 2149de7157224507bea78337507b471f (table test-table [id=014fafa350ce4528aff510dbd78b341e]) successfully deleted
I20260928 14:40:55.740815 4157 ts_tablet_manager.cc:1918] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:40:55.740870 4157 log.cc:1196] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/2149de7157224507bea78337507b471f
I20260928 14:40:55.741060 4157 ts_tablet_manager.cc:1939] T 2149de7157224507bea78337507b471f P f680208cf53c4dc7a8132fba43b1a877: Deleting consensus metadata
I20260928 14:40:55.741583 3559 catalog_manager.cc:4841] TS f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2:36279): tablet 2149de7157224507bea78337507b471f (table test-table [id=014fafa350ce4528aff510dbd78b341e]) successfully deleted
Sep 28 14:40:55 dist-test-slave-0n27 krb5kdc[3527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606455, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:40:55 dist-test-slave-0n27 krb5kdc[3527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606455, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:55.766376 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:55.754791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.1.201.62:39223)
0928 14:40:55.754988 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:55.754993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:55.755082 (+ 89us) server_negotiation.cc:407] Connection header received
0928 14:40:55.755163 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:55.755165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:55.755200 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:55.755266 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:55.755836 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:55.756375 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:55.757146 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:55.757355 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:55.757419 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:55.763823 (+ 6404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:55.763826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:55.763828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:55.763849 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:55.765003 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:55.765333 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:55.765335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:55.765336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:55.765402 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:55.765601 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:55.765603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:55.765604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:55.765754 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:55.765961 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:55.766171 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 14:40:55.766262 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260928 14:40:55.768488 3572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:40:55.768814 3572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:40:55.772756 3710 tablet_service.cc:1467] Processing CreateTablet for tablet e9d14f791aa74cea8930783d2c8b1397 (DEFAULT_TABLE table=test-table [id=d62089d1472244e4826878cd287d1add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:55.772806 3852 tablet_service.cc:1467] Processing CreateTablet for tablet e9d14f791aa74cea8930783d2c8b1397 (DEFAULT_TABLE table=test-table [id=d62089d1472244e4826878cd287d1add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:55.772886 3994 tablet_service.cc:1467] Processing CreateTablet for tablet e9d14f791aa74cea8930783d2c8b1397 (DEFAULT_TABLE table=test-table [id=d62089d1472244e4826878cd287d1add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:55.773041 3710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9d14f791aa74cea8930783d2c8b1397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:55.773059 3852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9d14f791aa74cea8930783d2c8b1397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:55.773101 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9d14f791aa74cea8930783d2c8b1397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:55.774901 4167 tablet_bootstrap.cc:492] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944: Bootstrap starting.
I20260928 14:40:55.775549 4169 tablet_bootstrap.cc:492] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965: Bootstrap starting.
I20260928 14:40:55.776016 4167 tablet_bootstrap.cc:654] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:55.776499 4169 tablet_bootstrap.cc:654] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:55.777108 4168 tablet_bootstrap.cc:492] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877: Bootstrap starting.
I20260928 14:40:55.777120 4167 tablet_bootstrap.cc:492] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944: No bootstrap required, opened a new log
I20260928 14:40:55.777207 4167 ts_tablet_manager.cc:1397] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:40:55.777590 4169 tablet_bootstrap.cc:492] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965: No bootstrap required, opened a new log
I20260928 14:40:55.777670 4169 ts_tablet_manager.cc:1397] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:55.777918 4167 raft_consensus.cc:348] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.778038 4167 raft_consensus.cc:374] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:55.778028 4168 tablet_bootstrap.cc:654] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:55.778074 4167 raft_consensus.cc:729] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf792a1e6849426bb8a219bc451df944, State: Initialized, Role: FOLLOWER
I20260928 14:40:55.778209 4167 consensus_queue.cc:260] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.778565 4167 ts_tablet_manager.cc:1428] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 14:40:55.778553 4169 raft_consensus.cc:348] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.778679 4169 raft_consensus.cc:374] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:55.778717 4169 raft_consensus.cc:729] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b12070c7de4dcfa6e6944c7df8f965, State: Initialized, Role: FOLLOWER
I20260928 14:40:55.778837 4169 consensus_queue.cc:260] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.779198 4169 ts_tablet_manager.cc:1428] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:55.779337 4168 tablet_bootstrap.cc:492] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877: No bootstrap required, opened a new log
I20260928 14:40:55.779435 4168 ts_tablet_manager.cc:1397] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:55.779790 4168 raft_consensus.cc:348] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.779898 4168 raft_consensus.cc:374] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:55.779930 4168 raft_consensus.cc:729] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f680208cf53c4dc7a8132fba43b1a877, State: Initialized, Role: FOLLOWER
I20260928 14:40:55.780037 4168 consensus_queue.cc:260] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.780625 4168 ts_tablet_manager.cc:1428] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:55.901080 4133 raft_consensus.cc:482] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:40:55.901196 4133 raft_consensus.cc:504] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.901549 4133 leader_election.cc:288] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919), 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337)
I20260928 14:40:55.901950 4013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d14f791aa74cea8930783d2c8b1397" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b12070c7de4dcfa6e6944c7df8f965" is_pre_election: true
I20260928 14:40:55.901952 3729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d14f791aa74cea8930783d2c8b1397" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf792a1e6849426bb8a219bc451df944" is_pre_election: true
I20260928 14:40:55.902056 4013 raft_consensus.cc:2454] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 0.
I20260928 14:40:55.902087 3729 raft_consensus.cc:2454] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 0.
I20260928 14:40:55.902380 3804 leader_election.cc:302] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b12070c7de4dcfa6e6944c7df8f965, f680208cf53c4dc7a8132fba43b1a877; no voters:
I20260928 14:40:55.902571 4133 raft_consensus.cc:2781] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:40:55.902628 4133 raft_consensus.cc:482] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:40:55.902653 4133 raft_consensus.cc:3037] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:55.903546 4133 raft_consensus.cc:504] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.903815 4133 leader_election.cc:288] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 election: Requested vote from peers bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919), 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337)
I20260928 14:40:55.904107 3729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d14f791aa74cea8930783d2c8b1397" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf792a1e6849426bb8a219bc451df944"
I20260928 14:40:55.904107 4013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d14f791aa74cea8930783d2c8b1397" candidate_uuid: "f680208cf53c4dc7a8132fba43b1a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b12070c7de4dcfa6e6944c7df8f965"
I20260928 14:40:55.904220 4013 raft_consensus.cc:3037] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:55.904220 3729 raft_consensus.cc:3037] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:55.905182 4013 raft_consensus.cc:2454] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 1.
I20260928 14:40:55.905194 3729 raft_consensus.cc:2454] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f680208cf53c4dc7a8132fba43b1a877 in term 1.
I20260928 14:40:55.905438 3806 leader_election.cc:302] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf792a1e6849426bb8a219bc451df944, f680208cf53c4dc7a8132fba43b1a877; no voters:
I20260928 14:40:55.905602 4133 raft_consensus.cc:2781] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:55.905700 4133 raft_consensus.cc:686] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 1 LEADER]: Becoming Leader. State: Replica: f680208cf53c4dc7a8132fba43b1a877, State: Running, Role: LEADER
I20260928 14:40:55.905844 4133 consensus_queue.cc:237] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.907308 3571 catalog_manager.cc:5495] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 reported cstate change: term changed from 0 to 1, leader changed from <none> to f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2). New cstate: current_term: 1 leader_uuid: "f680208cf53c4dc7a8132fba43b1a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } health_report { overall_health: UNKNOWN } } }
I20260928 14:40:55.934239 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:55.931537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.1.201.2:36279)
0928 14:40:55.931777 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:55.931783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:55.931793 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:40:55.931826 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:55.931827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:55.931862 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:55.931925 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:55.932113 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:55.932605 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:55.933395 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:55.933587 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:55.933656 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:55.933870 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:55.933931 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:55.933982 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:55.934052 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 14:40:55.934090 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":108,"threads_started":1}
I20260928 14:40:55.939163 3729 raft_consensus.cc:1261] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Refusing update from remote peer f680208cf53c4dc7a8132fba43b1a877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:55.939163 4013 raft_consensus.cc:1261] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Refusing update from remote peer f680208cf53c4dc7a8132fba43b1a877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:55.939486 4151 consensus_queue.cc:1035] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:55.939607 4133 consensus_queue.cc:1035] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:55.967993 3572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:55.968113 3572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:55.969892 3572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4: Sending DeleteTablet for 3 replicas of tablet e9d14f791aa74cea8930783d2c8b1397
I20260928 14:40:55.970328 3710 tablet_service.cc:1514] Processing DeleteTablet for tablet e9d14f791aa74cea8930783d2c8b1397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:55 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:53082
I20260928 14:40:55.970521 4156 tablet_replica.cc:330] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944: stopping tablet replica
I20260928 14:40:55.970559 3994 tablet_service.cc:1514] Processing DeleteTablet for tablet e9d14f791aa74cea8930783d2c8b1397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:55 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:34288
I20260928 14:40:55.970655 4156 raft_consensus.cc:2229] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:55.970721 4158 tablet_replica.cc:330] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965: stopping tablet replica
I20260928 14:40:55.970808 4156 raft_consensus.cc:2258] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:55.970808 4158 raft_consensus.cc:2229] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:55.970909 4158 raft_consensus.cc:2258] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:55.971205 3852 tablet_service.cc:1514] Processing DeleteTablet for tablet e9d14f791aa74cea8930783d2c8b1397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:55 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:33918
I20260928 14:40:55.971349 4157 tablet_replica.cc:330] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877: stopping tablet replica
I20260928 14:40:55.971429 4157 raft_consensus.cc:2229] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:40:55.971540 4157 raft_consensus.cc:2258] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:55.971804 4156 ts_tablet_manager.cc:1905] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:55.971977 4158 ts_tablet_manager.cc:1905] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:55.972007 3572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:40:55.972348 3572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:40:55.974231 4157 ts_tablet_manager.cc:1905] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:55.974261 4156 ts_tablet_manager.cc:1918] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:40:55.974327 4156 log.cc:1196] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/e9d14f791aa74cea8930783d2c8b1397
I20260928 14:40:55.974591 4156 ts_tablet_manager.cc:1939] T e9d14f791aa74cea8930783d2c8b1397 P bf792a1e6849426bb8a219bc451df944: Deleting consensus metadata
I20260928 14:40:55.975148 3560 catalog_manager.cc:4841] TS bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919): tablet e9d14f791aa74cea8930783d2c8b1397 (table test-table [id=d62089d1472244e4826878cd287d1add]) successfully deleted
I20260928 14:40:55.976359 4157 ts_tablet_manager.cc:1918] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:40:55.976426 4157 log.cc:1196] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/e9d14f791aa74cea8930783d2c8b1397
I20260928 14:40:55.976646 4157 ts_tablet_manager.cc:1939] T e9d14f791aa74cea8930783d2c8b1397 P f680208cf53c4dc7a8132fba43b1a877: Deleting consensus metadata
I20260928 14:40:55.977025 3710 tablet_service.cc:1467] Processing CreateTablet for tablet 2342208f539444d58b654a0c5f16e27d (DEFAULT_TABLE table=test-table [id=ec439d5696cf4c1fb2fddcd15a3a65c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:55.977254 4158 ts_tablet_manager.cc:1918] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:40:55.977267 3710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2342208f539444d58b654a0c5f16e27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:55.977334 4158 log.cc:1196] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/e9d14f791aa74cea8930783d2c8b1397
I20260928 14:40:55.977543 4158 ts_tablet_manager.cc:1939] T e9d14f791aa74cea8930783d2c8b1397 P 76b12070c7de4dcfa6e6944c7df8f965: Deleting consensus metadata
I20260928 14:40:55.979105 4167 tablet_bootstrap.cc:492] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944: Bootstrap starting.
I20260928 14:40:55.979303 3559 catalog_manager.cc:4841] TS f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2:36279): tablet e9d14f791aa74cea8930783d2c8b1397 (table test-table [id=d62089d1472244e4826878cd287d1add]) successfully deleted
I20260928 14:40:55.979787 3558 catalog_manager.cc:4841] TS 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337): tablet e9d14f791aa74cea8930783d2c8b1397 (table test-table [id=d62089d1472244e4826878cd287d1add]) successfully deleted
I20260928 14:40:55.980026 4167 tablet_bootstrap.cc:654] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:55.980077 3994 tablet_service.cc:1467] Processing CreateTablet for tablet 2342208f539444d58b654a0c5f16e27d (DEFAULT_TABLE table=test-table [id=ec439d5696cf4c1fb2fddcd15a3a65c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:55.980302 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2342208f539444d58b654a0c5f16e27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:55.981899 4169 tablet_bootstrap.cc:492] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965: Bootstrap starting.
I20260928 14:40:55.982254 3852 tablet_service.cc:1467] Processing CreateTablet for tablet 2342208f539444d58b654a0c5f16e27d (DEFAULT_TABLE table=test-table [id=ec439d5696cf4c1fb2fddcd15a3a65c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:40:55.982465 3852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2342208f539444d58b654a0c5f16e27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:55.982712 4169 tablet_bootstrap.cc:654] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:55.983762 4169 tablet_bootstrap.cc:492] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965: No bootstrap required, opened a new log
I20260928 14:40:55.983824 4169 ts_tablet_manager.cc:1397] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:40:55.984146 4169 raft_consensus.cc:348] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.984223 4169 raft_consensus.cc:374] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:55.984243 4169 raft_consensus.cc:729] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b12070c7de4dcfa6e6944c7df8f965, State: Initialized, Role: FOLLOWER
I20260928 14:40:55.984364 4169 consensus_queue.cc:260] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.984534 4168 tablet_bootstrap.cc:492] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877: Bootstrap starting.
I20260928 14:40:55.984597 4169 ts_tablet_manager.cc:1428] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:40:55.985175 4168 tablet_bootstrap.cc:654] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:55.986222 4168 tablet_bootstrap.cc:492] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877: No bootstrap required, opened a new log
I20260928 14:40:55.986312 4168 ts_tablet_manager.cc:1397] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:40:55.986716 4168 raft_consensus.cc:348] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.986867 4168 raft_consensus.cc:374] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:55.986917 4168 raft_consensus.cc:729] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f680208cf53c4dc7a8132fba43b1a877, State: Initialized, Role: FOLLOWER
I20260928 14:40:55.987046 4168 consensus_queue.cc:260] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.987306 4168 ts_tablet_manager.cc:1428] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:40:55.987991 4167 tablet_bootstrap.cc:492] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944: No bootstrap required, opened a new log
I20260928 14:40:55.988080 4167 ts_tablet_manager.cc:1397] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 14:40:55.988528 4167 raft_consensus.cc:348] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.988642 4167 raft_consensus.cc:374] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:55.988691 4167 raft_consensus.cc:729] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf792a1e6849426bb8a219bc451df944, State: Initialized, Role: FOLLOWER
I20260928 14:40:55.988803 4167 consensus_queue.cc:260] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:55.989045 4167 ts_tablet_manager.cc:1428] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:40:56.060715 4174 raft_consensus.cc:482] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:40:56.060812 4174 raft_consensus.cc:504] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:56.061703 4174 leader_election.cc:288] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919), f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2:36279)
I20260928 14:40:56.065204 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:56.062016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:59217 (local address 127.1.201.2:36279)
0928 14:40:56.062132 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:56.062138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:56.062344 (+ 206us) server_negotiation.cc:407] Connection header received
0928 14:40:56.062488 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:56.062493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:56.062536 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:56.062612 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:56.062863 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.063328 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.064329 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.064687 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.064792 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:56.064854 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:56.065035 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 14:40:56.065099 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:40:56.065225 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:56.061917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:36279 (local address 127.1.201.3:59217)
0928 14:40:56.062237 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:56.062251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:56.062382 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:56.062624 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:56.062627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:56.062641 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:56.062761 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:56.062766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.063431 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.063435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:56.064211 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:56.064219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.064794 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.064798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:56.064898 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:56.064939 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:40:56.065019 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:40:56.065083 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260928 14:40:56.065477 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:56.061917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:36919 (local address 127.1.201.3:36209)
0928 14:40:56.062410 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:56.062420 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:56.062527 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:56.062780 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:56.062782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:56.062798 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:40:56.062908 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:56.062913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.063617 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.063620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:56.064282 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:56.064289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.064913 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.064916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:56.065032 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:56.065057 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:40:56.065135 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:40:56.065204 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":110,"thread_start_us":152,"threads_started":1}
I20260928 14:40:56.065490 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:56.062075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:36209 (local address 127.1.201.1:36919)
0928 14:40:56.062315 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:56.062327 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:56.062493 (+ 166us) server_negotiation.cc:407] Connection header received
0928 14:40:56.062632 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:56.062635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:56.062685 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:56.062769 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:40:56.063015 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.063494 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.064399 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.064802 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.064912 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:56.064972 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:56.065150 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 14:40:56.065217 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":55,"threads_started":1}
I20260928 14:40:56.065654 3873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2342208f539444d58b654a0c5f16e27d" candidate_uuid: "76b12070c7de4dcfa6e6944c7df8f965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f680208cf53c4dc7a8132fba43b1a877" is_pre_election: true
I20260928 14:40:56.065908 3873 raft_consensus.cc:2454] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b12070c7de4dcfa6e6944c7df8f965 in term 0.
I20260928 14:40:56.065989 3729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2342208f539444d58b654a0c5f16e27d" candidate_uuid: "76b12070c7de4dcfa6e6944c7df8f965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf792a1e6849426bb8a219bc451df944" is_pre_election: true
I20260928 14:40:56.066126 3729 raft_consensus.cc:2454] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b12070c7de4dcfa6e6944c7df8f965 in term 0.
I20260928 14:40:56.066385 3947 leader_election.cc:302] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b12070c7de4dcfa6e6944c7df8f965, f680208cf53c4dc7a8132fba43b1a877; no voters:
I20260928 14:40:56.066620 4174 raft_consensus.cc:2781] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:40:56.066686 4174 raft_consensus.cc:482] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:40:56.066738 4174 raft_consensus.cc:3037] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:56.067632 4174 raft_consensus.cc:504] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:56.067950 4174 leader_election.cc:288] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [CANDIDATE]: Term 1 election: Requested vote from peers bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919), f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2:36279)
I20260928 14:40:56.068287 3729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2342208f539444d58b654a0c5f16e27d" candidate_uuid: "76b12070c7de4dcfa6e6944c7df8f965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf792a1e6849426bb8a219bc451df944"
I20260928 14:40:56.068331 3873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2342208f539444d58b654a0c5f16e27d" candidate_uuid: "76b12070c7de4dcfa6e6944c7df8f965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f680208cf53c4dc7a8132fba43b1a877"
I20260928 14:40:56.068408 3729 raft_consensus.cc:3037] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:56.068436 3873 raft_consensus.cc:3037] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:56.069212 3729 raft_consensus.cc:2454] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b12070c7de4dcfa6e6944c7df8f965 in term 1.
I20260928 14:40:56.069345 3873 raft_consensus.cc:2454] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b12070c7de4dcfa6e6944c7df8f965 in term 1.
I20260928 14:40:56.069499 3948 leader_election.cc:302] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b12070c7de4dcfa6e6944c7df8f965, bf792a1e6849426bb8a219bc451df944; no voters:
I20260928 14:40:56.069715 4174 raft_consensus.cc:2781] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:56.069909 4174 raft_consensus.cc:686] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 LEADER]: Becoming Leader. State: Replica: 76b12070c7de4dcfa6e6944c7df8f965, State: Running, Role: LEADER
I20260928 14:40:56.070142 4174 consensus_queue.cc:237] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } }
I20260928 14:40:56.072149 3572 catalog_manager.cc:5495] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3). New cstate: current_term: 1 leader_uuid: "76b12070c7de4dcfa6e6944c7df8f965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76b12070c7de4dcfa6e6944c7df8f965" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41337 } health_report { overall_health: HEALTHY } } }
I20260928 14:40:56.107506 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:56.104345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.1.201.3:41337)
0928 14:40:56.104580 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:56.104586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:56.104596 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:40:56.104649 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:56.104650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:56.104693 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:56.104780 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:56.104935 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.105368 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.106165 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.106387 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.106461 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:56.106676 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:56.107084 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:56.107166 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:56.107230 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 14:40:56.107281 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":121,"threads_started":1}
I20260928 14:40:56.115907 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:56.113099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:57085 (local address 127.1.201.2:36279)
0928 14:40:56.113231 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:56.113234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:56.113421 (+ 187us) server_negotiation.cc:407] Connection header received
0928 14:40:56.113511 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:56.113513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:56.113570 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:56.113628 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:56.113898 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.114349 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.115082 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.115261 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.115324 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:56.115551 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:56.115615 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:56.115687 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:56.115788 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 14:40:56.115825 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:40:56.115955 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:56.113048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:36279 (local address 127.1.201.3:57085)
0928 14:40:56.113351 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:56.113362 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:56.113436 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:56.113653 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:56.113655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:56.113660 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:40:56.113741 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:56.113745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.114432 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.114434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:56.114995 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:56.115002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.115342 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.115345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:56.115406 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:56.115438 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:40:56.115703 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:40:56.115707 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:40:56.115784 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:40:56.115851 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":138,"threads_started":1}
I20260928 14:40:56.120893 3729 raft_consensus.cc:1261] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Refusing update from remote peer 76b12070c7de4dcfa6e6944c7df8f965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:56.120993 3873 raft_consensus.cc:1261] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Refusing update from remote peer 76b12070c7de4dcfa6e6944c7df8f965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:56.121292 4174 consensus_queue.cc:1035] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf792a1e6849426bb8a219bc451df944" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:56.121454 4187 consensus_queue.cc:1035] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f680208cf53c4dc7a8132fba43b1a877" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:56.137257 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:56.133961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:58041 (local address 127.1.201.3:41337)
0928 14:40:56.134055 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:56.134059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:56.134482 (+ 423us) server_negotiation.cc:407] Connection header received
0928 14:40:56.134591 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:56.134594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:56.134632 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:56.134715 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:56.134936 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.135400 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.136214 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.136467 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.136569 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:56.136835 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:56.136908 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:56.136999 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:56.137087 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 14:40:56.137129 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:40:56.137287 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:56.133906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:41337 (local address 127.1.201.2:58041)
0928 14:40:56.134363 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:56.134384 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:56.134515 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:56.134734 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:56.134737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:56.134744 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:40:56.134832 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:56.134839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.135503 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.135507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:56.136096 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:56.136104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:56.136579 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:56.136583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:56.136669 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:56.136699 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:40:56.137014 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:40:56.137020 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:40:56.137079 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:40:56.137151 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":84,"threads_started":1}
W20260928 14:40:56.137931 4139 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:40:57.690196 3571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:57.690326 3571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:40:57.692488 3571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05f83e020b0c4da0b3ce6332360705c4: Sending DeleteTablet for 3 replicas of tablet 2342208f539444d58b654a0c5f16e27d
I20260928 14:40:57.692921 3710 tablet_service.cc:1514] Processing DeleteTablet for tablet 2342208f539444d58b654a0c5f16e27d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:57 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:53082
I20260928 14:40:57.693027 3852 tablet_service.cc:1514] Processing DeleteTablet for tablet 2342208f539444d58b654a0c5f16e27d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:57 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:33918
I20260928 14:40:57.693252 4221 tablet_replica.cc:330] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944: stopping tablet replica
I20260928 14:40:57.693323 3994 tablet_service.cc:1514] Processing DeleteTablet for tablet 2342208f539444d58b654a0c5f16e27d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:40:57 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:34288
I20260928 14:40:57.693346 4222 tablet_replica.cc:330] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877: stopping tablet replica
I20260928 14:40:57.693375 4221 raft_consensus.cc:2229] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:57.693459 4222 raft_consensus.cc:2229] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:40:57.693508 4221 raft_consensus.cc:2258] T 2342208f539444d58b654a0c5f16e27d P bf792a1e6849426bb8a219bc451df944 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:57.693583 4223 tablet_replica.cc:330] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965: stopping tablet replica
I20260928 14:40:57.693594 4222 raft_consensus.cc:2258] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:57.693745 4223 raft_consensus.cc:2229] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:40:57.694048 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 3645
I20260928 14:40:57.695161 4222 ts_tablet_manager.cc:1905] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:57.694089 4223 raft_consensus.cc:2258] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:40:57.696653 4223 ts_tablet_manager.cc:1905] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:40:57.698063 4222 ts_tablet_manager.cc:1918] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:40:57.698146 4222 log.cc:1196] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/2342208f539444d58b654a0c5f16e27d
I20260928 14:40:57.698465 4222 ts_tablet_manager.cc:1939] T 2342208f539444d58b654a0c5f16e27d P f680208cf53c4dc7a8132fba43b1a877: Deleting consensus metadata
I20260928 14:40:57.699102 3559 catalog_manager.cc:4841] TS f680208cf53c4dc7a8132fba43b1a877 (127.1.201.2:36279): tablet 2342208f539444d58b654a0c5f16e27d (table test-table [id=ec439d5696cf4c1fb2fddcd15a3a65c6]) successfully deleted
I20260928 14:40:57.699340 4223 ts_tablet_manager.cc:1918] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:40:57.699399 4223 log.cc:1196] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/2342208f539444d58b654a0c5f16e27d
I20260928 14:40:57.699657 4223 ts_tablet_manager.cc:1939] T 2342208f539444d58b654a0c5f16e27d P 76b12070c7de4dcfa6e6944c7df8f965: Deleting consensus metadata
I20260928 14:40:57.700260 3558 catalog_manager.cc:4841] TS 76b12070c7de4dcfa6e6944c7df8f965 (127.1.201.3:41337): tablet 2342208f539444d58b654a0c5f16e27d (table test-table [id=ec439d5696cf4c1fb2fddcd15a3a65c6]) successfully deleted
W20260928 14:40:57.701267 3560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:36919 (error 108)
I20260928 14:40:57.701853 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 3787
I20260928 14:40:57.702711 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:57.701949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:36919 (local address 127.0.0.1:55388)
0928 14:40:57.702568 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:57.702628 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":71,"threads_started":1}
W20260928 14:40:57.702864 3560 catalog_manager.cc:4584] TS bf792a1e6849426bb8a219bc451df944 (127.1.201.1:36919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2342208f539444d58b654a0c5f16e27d: Network error: Client connection negotiation failed: client connection to 127.1.201.1:36919: connect: Connection refused (error 111)
I20260928 14:40:57.709992 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 3929
I20260928 14:40:57.716310 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 3541
2026-09-28T14:40:57Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6004 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:57 dist-test-slave-0n27 krb5kdc[4228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:57 dist-test-slave-0n27 krb5kdc[4228](info): set up 2 sockets
Sep 28 14:40:57 dist-test-slave-0n27 krb5kdc[4228](info): commencing operation
krb5kdc: starting...
W20260928 14:40:58.702492 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:40:58 dist-test-slave-0n27 krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:40:58.811918 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:38623
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:58.910929 4241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:58.911172 4241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:58.911237 4241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:58.914527 4241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:58.914590 4241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:58.914623 4241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:58.914669 4241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:58.914702 4241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:58.919312 4241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:58.920167 4241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:58.921011 4241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:58.928380 4247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:58.928376 4246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:58.928432 4241 server_base.cc:1034] running on GCE node
W20260928 14:40:58.928380 4249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:58.929256 4241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:58.930146 4241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:58.931301 4241 hybrid_clock.cc:648] HybridClock initialized: now 1790606458931271 us; error 41 us; skew 500 ppm
Sep 28 14:40:58 dist-test-slave-0n27 krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:58.933578 4241 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:40:58.934324 4241 webserver.cc:460] Webserver started at http://127.1.201.62:35505/ using document root <none> and password file <none>
I20260928 14:40:58.934798 4241 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:58.934852 4241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:58.935052 4241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:58.936810 4241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "eaa8a96a09fa4c86acb9a0680606790c"
format_stamp: "Formatted at 2026-09-28 14:40:58 on dist-test-slave-0n27"
server_key: "c373d22352f5815f9b4774b1d74a3e96"
server_key_iv: "8006e25cf58411b4e9d51d862c058290"
server_key_version: "encryptionkey@0"
I20260928 14:40:58.937325 4241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "eaa8a96a09fa4c86acb9a0680606790c"
format_stamp: "Formatted at 2026-09-28 14:40:58 on dist-test-slave-0n27"
server_key: "c373d22352f5815f9b4774b1d74a3e96"
server_key_iv: "8006e25cf58411b4e9d51d862c058290"
server_key_version: "encryptionkey@0"
I20260928 14:40:58.940486 4241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:40:58.942545 4256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:58.943549 4241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:58.943668 4241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "eaa8a96a09fa4c86acb9a0680606790c"
format_stamp: "Formatted at 2026-09-28 14:40:58 on dist-test-slave-0n27"
server_key: "c373d22352f5815f9b4774b1d74a3e96"
server_key_iv: "8006e25cf58411b4e9d51d862c058290"
server_key_version: "encryptionkey@0"
I20260928 14:40:58.943783 4241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:58.969250 4241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:58.971807 4241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:58.971987 4241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:58.979056 4241 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:38623
I20260928 14:40:58.979079 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:38623 every 8 connection(s)
I20260928 14:40:58.979813 4241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:58.982014 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:58.986745 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 4241
I20260928 14:40:58.986775 4309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c: Bootstrap starting.
I20260928 14:40:58.986832 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:40:58.987082 1828 external_mini_cluster.cc:1428] Setting key e959f80978dfab75b16d5e9bfd6014bc
I20260928 14:40:58.989094 4309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:58.989945 4309 log.cc:824] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:58.991786 4309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c: No bootstrap required, opened a new log
I20260928 14:40:58.995281 4309 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa8a96a09fa4c86acb9a0680606790c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38623 } }
Sep 28 14:40:58 dist-test-slave-0n27 krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:58.995503 4309 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:58.995581 4309 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa8a96a09fa4c86acb9a0680606790c, State: Initialized, Role: FOLLOWER
I20260928 14:40:58.996024 4309 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa8a96a09fa4c86acb9a0680606790c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38623 } }
I20260928 14:40:58.996179 4309 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:58.996259 4309 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:58.996387 4309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:58.997507 4309 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa8a96a09fa4c86acb9a0680606790c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38623 } }
I20260928 14:40:58.997876 4309 leader_election.cc:302] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa8a96a09fa4c86acb9a0680606790c; no voters:
I20260928 14:40:58.998205 4309 leader_election.cc:288] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:58.998291 4314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:58.998487 4314 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [term 1 LEADER]: Becoming Leader. State: Replica: eaa8a96a09fa4c86acb9a0680606790c, State: Running, Role: LEADER
I20260928 14:40:58.998814 4314 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa8a96a09fa4c86acb9a0680606790c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38623 } }
I20260928 14:40:58.999256 4309 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:59.000505 4316 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaa8a96a09fa4c86acb9a0680606790c. Latest consensus state: current_term: 1 leader_uuid: "eaa8a96a09fa4c86acb9a0680606790c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa8a96a09fa4c86acb9a0680606790c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38623 } } }
I20260928 14:40:59.000882 4316 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:59.001127 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:58.988265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.1.201.62:38623)
0928 14:40:58.988639 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:58.988658 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:58.988695 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:40:58.989490 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:58.989510 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:58.989907 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:58.990240 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:58.991005 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:58.992288 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:58.993028 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:58.993651 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:58.993826 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:58.996037 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:58.996044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:58.996058 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:58.996086 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:58.998058 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:58.998553 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:58.998560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:58.998566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:58.998706 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:58.998972 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:58.998975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:58.998976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:58.999406 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:58.999584 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:58.999915 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 14:40:59.000208 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260928 14:40:59.000795 4315 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaa8a96a09fa4c86acb9a0680606790c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa8a96a09fa4c86acb9a0680606790c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38623 } } }
I20260928 14:40:59.001518 4315 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c [sys.catalog]: This master's current role is: LEADER
W20260928 14:40:59.003963 4328 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:59.004021 4328 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:59.004311 4331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:59.004930 4331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:59.009229 4331 catalog_manager.cc:1347] Generated new cluster ID: a15ce0dec97f4355a0ffa7855ad43b98
I20260928 14:40:59.009282 4331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:59.014355 4331 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:59.015311 4331 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:59.028494 4331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaa8a96a09fa4c86acb9a0680606790c: Generated new TSK 0
I20260928 14:40:59.028982 4331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:40:59.060045 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38623
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:59.159737 4335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:59.159943 4335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:59.159979 4335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:59.163406 4335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:59.163444 4335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:59.163493 4335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:40:59.168180 4335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:59.169020 4335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:59.169898 4335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:59.178232 4340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:59.178277 4343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:59.178287 4341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:59.178440 4335 server_base.cc:1034] running on GCE node
I20260928 14:40:59.178789 4335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:59.179332 4335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:59.180491 4335 hybrid_clock.cc:648] HybridClock initialized: now 1790606459180473 us; error 26 us; skew 500 ppm
Sep 28 14:40:59 dist-test-slave-0n27 krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:59.182909 4335 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:40:59.183676 4335 webserver.cc:460] Webserver started at http://127.1.201.1:33925/ using document root <none> and password file <none>
I20260928 14:40:59.184198 4335 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:59.184254 4335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:59.184465 4335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:59.186252 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "3b6e52d7a5ef41858bef68e3147b2a20"
format_stamp: "Formatted at 2026-09-28 14:40:59 on dist-test-slave-0n27"
server_key: "12f8e63722a096766a9e5f1dbf24dae1"
server_key_iv: "9e0dbbb3c5c197fc21fb8fe143e73be0"
server_key_version: "encryptionkey@0"
I20260928 14:40:59.186771 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "3b6e52d7a5ef41858bef68e3147b2a20"
format_stamp: "Formatted at 2026-09-28 14:40:59 on dist-test-slave-0n27"
server_key: "12f8e63722a096766a9e5f1dbf24dae1"
server_key_iv: "9e0dbbb3c5c197fc21fb8fe143e73be0"
server_key_version: "encryptionkey@0"
I20260928 14:40:59.189990 4335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:40:59.192088 4350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:59.193159 4335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:59.193274 4335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "3b6e52d7a5ef41858bef68e3147b2a20"
format_stamp: "Formatted at 2026-09-28 14:40:59 on dist-test-slave-0n27"
server_key: "12f8e63722a096766a9e5f1dbf24dae1"
server_key_iv: "9e0dbbb3c5c197fc21fb8fe143e73be0"
server_key_version: "encryptionkey@0"
I20260928 14:40:59.193362 4335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:59.211763 4335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:59.214267 4335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:59.214428 4335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:59.214908 4335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:40:59.215590 4335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:59.215656 4335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:59.215726 4335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:59.215757 4335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:59.223388 4335 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:35401
I20260928 14:40:59.223407 4464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:35401 every 8 connection(s)
I20260928 14:40:59.224256 4335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:40:59.224668 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 4335
I20260928 14:40:59.224789 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:40:59.225087 1828 external_mini_cluster.cc:1428] Setting key 38d2cc1d088abc5c40b47537950ef0cb
Sep 28 14:40:59 dist-test-slave-0n27 krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
I20260928 14:40:59.237212 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:59.226296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:50309 (local address 127.1.201.62:38623)
0928 14:40:59.226450 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:59.226454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:59.226896 (+ 442us) server_negotiation.cc:407] Connection header received
0928 14:40:59.228156 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:59.228159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:59.228237 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:59.228328 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:59.229324 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.229876 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.230861 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.231091 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.231192 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:59.233636 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:59.233639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:59.233640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:59.233661 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:59.235051 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:59.235501 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:59.235504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:59.235505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:59.235572 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:59.235949 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:59.235952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:59.235953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:59.236111 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:59.236190 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:59.236909 (+ 719us) server_negotiation.cc:299] Negotiation successful
0928 14:40:59.237032 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 14:40:59.237972 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:59.226292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38623 (local address 127.1.201.1:50309)
0928 14:40:59.226773 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:59.226809 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:59.227969 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:59.228488 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:59.228494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:59.228799 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:59.229196 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:59.229207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.229981 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.229984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:59.230752 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:59.230760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.231225 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.231229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:59.231368 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:59.231838 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:59.231854 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:59.233496 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:59.235161 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:59.235165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:59.235175 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:59.235420 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:59.235691 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:59.235695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:59.235697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:59.235869 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:59.236267 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:59.236272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:59.236602 (+ 330us) client_negotiation.cc:769] Sending connection context
0928 14:40:59.236847 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 14:40:59.237140 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":137,"threads_started":1}
I20260928 14:40:59.239078 4465 heartbeater.cc:347] Connected to a master server at 127.1.201.62:38623
I20260928 14:40:59.239290 4465 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:59.239742 4465 heartbeater.cc:511] Master 127.1.201.62:38623 requested a full tablet report, sending...
I20260928 14:40:59.241132 4273 ts_manager.cc:194] Registered new tserver with Master: 3b6e52d7a5ef41858bef68e3147b2a20 (127.1.201.1:35401)
I20260928 14:40:59.242223 4273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:50309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:40:59.250738 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38623
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:59.352183 4470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:59.352370 4470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:59.352428 4470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:59.356244 4470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:59.356293 4470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:59.356382 4470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:40:59.361104 4470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:59.361900 4470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:59.362748 4470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:59.371469 4476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:59.371481 4478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:59.371477 4475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:59.371855 4470 server_base.cc:1034] running on GCE node
I20260928 14:40:59.372196 4470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:59.372758 4470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:59.373898 4470 hybrid_clock.cc:648] HybridClock initialized: now 1790606459373887 us; error 21 us; skew 500 ppm
Sep 28 14:40:59 dist-test-slave-0n27 krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:59.376431 4470 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:40:59.377159 4470 webserver.cc:460] Webserver started at http://127.1.201.2:38395/ using document root <none> and password file <none>
I20260928 14:40:59.377657 4470 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:59.377714 4470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:59.377959 4470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:59.379778 4470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "b9918ab2f7a54aed9f05095126780feb"
format_stamp: "Formatted at 2026-09-28 14:40:59 on dist-test-slave-0n27"
server_key: "b8e843f42c183885083e011c8c9ffd6c"
server_key_iv: "a05b926eba6e945b51a281ed76a926e9"
server_key_version: "encryptionkey@0"
I20260928 14:40:59.380286 4470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "b9918ab2f7a54aed9f05095126780feb"
format_stamp: "Formatted at 2026-09-28 14:40:59 on dist-test-slave-0n27"
server_key: "b8e843f42c183885083e011c8c9ffd6c"
server_key_iv: "a05b926eba6e945b51a281ed76a926e9"
server_key_version: "encryptionkey@0"
I20260928 14:40:59.383548 4470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:40:59.386037 4485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:59.387118 4470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:59.387236 4470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "b9918ab2f7a54aed9f05095126780feb"
format_stamp: "Formatted at 2026-09-28 14:40:59 on dist-test-slave-0n27"
server_key: "b8e843f42c183885083e011c8c9ffd6c"
server_key_iv: "a05b926eba6e945b51a281ed76a926e9"
server_key_version: "encryptionkey@0"
I20260928 14:40:59.387331 4470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:59.400236 4470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:59.402549 4470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:59.402738 4470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:59.403223 4470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:40:59.403893 4470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:59.403934 4470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:59.403990 4470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:59.404021 4470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:59.411863 4470 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:43277
I20260928 14:40:59.411886 4598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:43277 every 8 connection(s)
I20260928 14:40:59.412580 4470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:40:59.415503 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 4470
I20260928 14:40:59.415617 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:40:59.415879 1828 external_mini_cluster.cc:1428] Setting key 92c269de063212af22142b36a6b5d746
Sep 28 14:40:59 dist-test-slave-0n27 krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:59.425177 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:59.414024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:46647 (local address 127.1.201.62:38623)
0928 14:40:59.414160 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:59.414164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:59.414793 (+ 629us) server_negotiation.cc:407] Connection header received
0928 14:40:59.415972 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:59.415975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:59.416028 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:59.416114 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:59.417435 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.418178 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.419184 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.419533 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.419642 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:59.421963 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:59.421966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:59.421968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:59.421988 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:59.423303 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:59.423838 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:59.423841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:59.423842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:59.423903 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:59.424255 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:59.424258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:59.424259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:59.424393 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:59.424457 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:59.424995 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 14:40:59.425075 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
I20260928 14:40:59.426105 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:59.414322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38623 (local address 127.1.201.2:46647)
0928 14:40:59.414669 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:59.414703 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:59.415786 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:59.416261 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:59.416268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:59.416702 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:59.417291 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:59.417310 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.418286 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.418289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:59.419079 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:59.419088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.419654 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.419656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:59.419774 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:59.420238 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:59.420252 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:59.421823 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:59.423408 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:59.423415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:59.423429 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:59.423751 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:59.423986 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:59.423991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:59.423993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:59.424173 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:59.424484 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:59.424492 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:59.424774 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 14:40:59.424975 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 14:40:59.425249 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260928 14:40:59.427050 4599 heartbeater.cc:347] Connected to a master server at 127.1.201.62:38623
I20260928 14:40:59.427282 4599 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:59.427685 4599 heartbeater.cc:511] Master 127.1.201.62:38623 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
I20260928 14:40:59.428633 4273 ts_manager.cc:194] Registered new tserver with Master: b9918ab2f7a54aed9f05095126780feb (127.1.201.2:43277)
I20260928 14:40:59.429196 4273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:46647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:40:59.440495 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38623
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:59.539788 4604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:59.540022 4604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:59.540078 4604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:59.543818 4604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:59.543890 4604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:59.543974 4604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:40:59.549036 4604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:59.549961 4604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:59.550843 4604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:59.559654 4609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:59.559718 4610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:59.560137 4612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:59.560415 4604 server_base.cc:1034] running on GCE node
I20260928 14:40:59.560819 4604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:59.561498 4604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:59.564749 4604 hybrid_clock.cc:648] HybridClock initialized: now 1790606459564712 us; error 43 us; skew 500 ppm
Sep 28 14:40:59 dist-test-slave-0n27 krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:59.568017 4604 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:40:59.568797 4604 webserver.cc:460] Webserver started at http://127.1.201.3:46033/ using document root <none> and password file <none>
I20260928 14:40:59.569242 4604 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:59.569293 4604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:59.569489 4604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:59.571326 4604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "ca3499e054af42af995ff1e7c4e082fb"
format_stamp: "Formatted at 2026-09-28 14:40:59 on dist-test-slave-0n27"
server_key: "b174e097da02f8611fc053af6903cecc"
server_key_iv: "06594458c0dbbe2d80097993a89fbcb4"
server_key_version: "encryptionkey@0"
I20260928 14:40:59.571762 4604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "ca3499e054af42af995ff1e7c4e082fb"
format_stamp: "Formatted at 2026-09-28 14:40:59 on dist-test-slave-0n27"
server_key: "b174e097da02f8611fc053af6903cecc"
server_key_iv: "06594458c0dbbe2d80097993a89fbcb4"
server_key_version: "encryptionkey@0"
I20260928 14:40:59.575027 4604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:40:59.577181 4619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:59.578366 4604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:59.578490 4604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "ca3499e054af42af995ff1e7c4e082fb"
format_stamp: "Formatted at 2026-09-28 14:40:59 on dist-test-slave-0n27"
server_key: "b174e097da02f8611fc053af6903cecc"
server_key_iv: "06594458c0dbbe2d80097993a89fbcb4"
server_key_version: "encryptionkey@0"
I20260928 14:40:59.578588 4604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:59.597728 4604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:59.600409 4604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:59.600574 4604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:59.601035 4604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:40:59.601675 4604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:40:59.601723 4604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:59.601775 4604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:40:59.601863 4604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:59.609676 4604 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:38569
I20260928 14:40:59.609743 4732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:38569 every 8 connection(s)
I20260928 14:40:59.610483 4604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:40:59.615531 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 4604
I20260928 14:40:59.615619 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:40:59.615875 1828 external_mini_cluster.cc:1428] Setting key 9b5ecabdf028d24b35ea79854329e4e6
Sep 28 14:40:59 dist-test-slave-0n27 krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:40:59.622258 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:59.611971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:49625 (local address 127.1.201.62:38623)
0928 14:40:59.612127 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:59.612131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:59.612809 (+ 678us) server_negotiation.cc:407] Connection header received
0928 14:40:59.614077 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:59.614080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:59.614142 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:59.614227 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:59.615312 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.615894 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.617244 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.617469 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.617582 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:59.619667 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:59.619670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:59.619672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:59.619692 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:59.620739 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:59.621147 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:59.621149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:59.621150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:59.621215 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:59.621451 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:59.621453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:59.621454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:59.621590 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:59.621657 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:59.622067 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 14:40:59.622158 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:40:59.622884 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:59.612308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38623 (local address 127.1.201.3:49625)
0928 14:40:59.612688 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:59.612723 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:59.613882 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:59.614433 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:59.614440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:59.614765 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:59.615155 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:59.615167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.616024 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.616027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:59.617124 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:59.617134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:59.617631 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:59.617635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:59.617841 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:59.618294 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:59.618308 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:59.619538 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:59.620835 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:59.620838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:59.620847 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:59.621059 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:59.621282 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:59.621284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:59.621285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:59.621387 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:59.621664 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:59.621668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:59.621885 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 14:40:59.622011 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:40:59.622233 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260928 14:40:59.623849 4733 heartbeater.cc:347] Connected to a master server at 127.1.201.62:38623
I20260928 14:40:59.624040 4733 heartbeater.cc:464] Registering TS with master...
I20260928 14:40:59.624384 4733 heartbeater.cc:511] Master 127.1.201.62:38623 requested a full tablet report, sending...
I20260928 14:40:59.625166 4273 ts_manager.cc:194] Registered new tserver with Master: ca3499e054af42af995ff1e7c4e082fb (127.1.201.3:38569)
I20260928 14:40:59.625653 4273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:49625
I20260928 14:40:59.629038 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:40:59.640312 4312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:59.639155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.1.201.62:38623)
0928 14:40:59.639258 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:59.639262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:59.639413 (+ 151us) server_negotiation.cc:407] Connection header received
0928 14:40:59.639541 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:59.639543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:59.639581 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:59.639654 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:59.640240 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33680: BlockingRecv error: recv got EOF from 127.0.0.1:33680 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:40:59.641244 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 4335
I20260928 14:40:59.646520 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 4470
I20260928 14:40:59.651340 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 4604
I20260928 14:40:59.656033 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 4241
2026-09-28T14:40:59Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1938 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:59 dist-test-slave-0n27 krb5kdc[4745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:59 dist-test-slave-0n27 krb5kdc[4745](info): set up 2 sockets
Sep 28 14:40:59 dist-test-slave-0n27 krb5kdc[4745](info): commencing operation
krb5kdc: starting...
W20260928 14:41:00.638355 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:41:00 dist-test-slave-0n27 krb5kdc[4745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:41:00.748971 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42511
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:00.847184 4758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:00.847383 4758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:00.847419 4758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:00.851029 4758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:00.851070 4758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:00.851089 4758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:00.851125 4758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:00.851155 4758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:00.855774 4758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:42511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:00.856608 4758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:00.857463 4758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:00.864748 4763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:00.864753 4766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:00.864817 4758 server_base.cc:1034] running on GCE node
W20260928 14:41:00.864765 4764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:00.865370 4758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:00.866277 4758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:00.867442 4758 hybrid_clock.cc:648] HybridClock initialized: now 1790606460867430 us; error 23 us; skew 500 ppm
Sep 28 14:41:00 dist-test-slave-0n27 krb5kdc[4745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:00.873724 4758 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:41:00.874511 4758 webserver.cc:460] Webserver started at http://127.1.201.62:35073/ using document root <none> and password file <none>
I20260928 14:41:00.874955 4758 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:00.875031 4758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:00.875222 4758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:00.876892 4758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "5e70c6c293d142ec8b349e3a8ce14ca4"
format_stamp: "Formatted at 2026-09-28 14:41:00 on dist-test-slave-0n27"
server_key: "40ac28036bd05d2e0b1424514b76c43d"
server_key_iv: "ec8f3b58ece311575da519409c58b150"
server_key_version: "encryptionkey@0"
I20260928 14:41:00.877398 4758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "5e70c6c293d142ec8b349e3a8ce14ca4"
format_stamp: "Formatted at 2026-09-28 14:41:00 on dist-test-slave-0n27"
server_key: "40ac28036bd05d2e0b1424514b76c43d"
server_key_iv: "ec8f3b58ece311575da519409c58b150"
server_key_version: "encryptionkey@0"
I20260928 14:41:00.880491 4758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260928 14:41:00.882521 4773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:00.883539 4758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:00.883672 4758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "5e70c6c293d142ec8b349e3a8ce14ca4"
format_stamp: "Formatted at 2026-09-28 14:41:00 on dist-test-slave-0n27"
server_key: "40ac28036bd05d2e0b1424514b76c43d"
server_key_iv: "ec8f3b58ece311575da519409c58b150"
server_key_version: "encryptionkey@0"
I20260928 14:41:00.883767 4758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:00.898123 4758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:00.900440 4758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:00.900611 4758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:00.906840 4758 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:42511
I20260928 14:41:00.907083 4825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:42511 every 8 connection(s)
I20260928 14:41:00.907657 4758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:00.910046 4826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:00.913853 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 4758
I20260928 14:41:00.913949 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:00.914165 1828 external_mini_cluster.cc:1428] Setting key 6a86022941fa7704213e0e7b615cee17
I20260928 14:41:00.914804 4826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4: Bootstrap starting.
I20260928 14:41:00.916829 4826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:00.917724 4826 log.cc:824] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:00.919443 4826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4: No bootstrap required, opened a new log
Sep 28 14:41:00 dist-test-slave-0n27 krb5kdc[4745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:00.923166 4826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e70c6c293d142ec8b349e3a8ce14ca4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42511 } }
I20260928 14:41:00.923401 4826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:00.923466 4826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e70c6c293d142ec8b349e3a8ce14ca4, State: Initialized, Role: FOLLOWER
I20260928 14:41:00.923849 4826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e70c6c293d142ec8b349e3a8ce14ca4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42511 } }
I20260928 14:41:00.923985 4826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:00.924062 4826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:00.924165 4826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:00.924897 4826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e70c6c293d142ec8b349e3a8ce14ca4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42511 } }
I20260928 14:41:00.925225 4826 leader_election.cc:302] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e70c6c293d142ec8b349e3a8ce14ca4; no voters:
I20260928 14:41:00.925519 4826 leader_election.cc:288] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:00.925612 4831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:00.925827 4831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e70c6c293d142ec8b349e3a8ce14ca4, State: Running, Role: LEADER
I20260928 14:41:00.926126 4831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e70c6c293d142ec8b349e3a8ce14ca4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42511 } }
I20260928 14:41:00.926729 4826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:00.927119 4831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e70c6c293d142ec8b349e3a8ce14ca4. Latest consensus state: current_term: 1 leader_uuid: "5e70c6c293d142ec8b349e3a8ce14ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e70c6c293d142ec8b349e3a8ce14ca4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42511 } } }
I20260928 14:41:00.927338 4831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:00.927518 4832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e70c6c293d142ec8b349e3a8ce14ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e70c6c293d142ec8b349e3a8ce14ca4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42511 } } }
I20260928 14:41:00.927671 4832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:00.927798 4838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:00.927779 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:00.915248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.1.201.62:42511)
0928 14:41:00.915585 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:00.915606 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:00.915634 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:41:00.916199 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:00.916213 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:00.916491 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:00.916758 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:00.917630 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:00.918716 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:00.919744 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:00.920133 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:00.920234 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:00.922592 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:00.922596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:00.922607 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:00.922629 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:00.924301 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:00.924699 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:00.924704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:00.924709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:00.924811 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:00.925095 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:00.925099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:00.925100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:00.925581 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:00.926413 (+ 832us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:00.926565 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 14:41:00.926827 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":119,"threads_started":1}
I20260928 14:41:00.930500 4838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:00.931205 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:41:00.935261 4838 catalog_manager.cc:1347] Generated new cluster ID: d84048382abf4c6fba7ec6daca40a094
I20260928 14:41:00.935309 4838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:00.949496 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:00.944595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.1.201.62:42511)
0928 14:41:00.944696 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:00.944700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:00.944937 (+ 237us) server_negotiation.cc:407] Connection header received
0928 14:41:00.945025 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:00.945028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:00.945070 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:00.945140 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:00.945585 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:00.946346 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:00.947443 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:00.947835 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:00.947928 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:00.948679 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:00.948683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:00.948686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:00.949121 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:00.949149 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:00.949165 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:00.949246 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:00.949335 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 14:41:00.949381 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:41:00.950673 4838 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:00.951390 4838 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:00.957615 4838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e70c6c293d142ec8b349e3a8ce14ca4: Generated new TSK 0
I20260928 14:41:00.958215 4838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:41:00.961738 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 4758
2026-09-28T14:41:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1304 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:41:01 dist-test-slave-0n27 krb5kdc[4859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:41:01 dist-test-slave-0n27 krb5kdc[4859](info): set up 2 sockets
Sep 28 14:41:01 dist-test-slave-0n27 krb5kdc[4859](info): commencing operation
krb5kdc: starting...
W20260928 14:41:01.942234 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:41:02.050786 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:43069
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:02.150696 4872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:02.150890 4872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:02.150923 4872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:02.154346 4872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:02.154386 4872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:02.154402 4872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:02.154417 4872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:02.154428 4872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:02.159173 4872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:43069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:02.159916 4872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:02.160696 4872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:02.167935 4878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:02.168000 4877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:02.168027 4872 server_base.cc:1034] running on GCE node
W20260928 14:41:02.167945 4880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:02.168640 4872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:02.169549 4872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:02.170697 4872 hybrid_clock.cc:648] HybridClock initialized: now 1790606462170676 us; error 35 us; skew 500 ppm
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:02.173118 4872 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:41:02.173854 4872 webserver.cc:460] Webserver started at http://127.1.201.62:37643/ using document root <none> and password file <none>
I20260928 14:41:02.174342 4872 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:02.174403 4872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:02.174552 4872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:02.176242 4872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "c88bc85a0bce498b9d10c45227407e5b"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "e956ca0a3d239dd1018937de56914808"
server_key_iv: "415b9df5eabe6e4e8be4886061b35baf"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.176718 4872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "c88bc85a0bce498b9d10c45227407e5b"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "e956ca0a3d239dd1018937de56914808"
server_key_iv: "415b9df5eabe6e4e8be4886061b35baf"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.180351 4872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:02.182600 4887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:02.183722 4872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 14:41:02.183842 4872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "c88bc85a0bce498b9d10c45227407e5b"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "e956ca0a3d239dd1018937de56914808"
server_key_iv: "415b9df5eabe6e4e8be4886061b35baf"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.183946 4872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:02.191856 4872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:02.194267 4872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:02.194434 4872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:02.200557 4872 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:43069
I20260928 14:41:02.200807 4939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:43069 every 8 connection(s)
I20260928 14:41:02.201361 4872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:02.203811 4940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:02.205502 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 4872
I20260928 14:41:02.205629 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:02.205937 1828 external_mini_cluster.cc:1428] Setting key c37ce0201709b7fb2ba31df47cbb6222
I20260928 14:41:02.210110 4940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b: Bootstrap starting.
I20260928 14:41:02.212423 4940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:02.213215 4940 log.cc:824] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:02.215135 4940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b: No bootstrap required, opened a new log
I20260928 14:41:02.219601 4940 raft_consensus.cc:348] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88bc85a0bce498b9d10c45227407e5b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43069 } }
I20260928 14:41:02.219780 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:02.207201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.1.201.62:43069)
0928 14:41:02.207578 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:02.207596 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:02.207635 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:41:02.208355 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:02.208376 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:02.208753 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:02.209068 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:02.209742 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.211160 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.211956 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.212575 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.212734 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:02.215112 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:02.215118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:02.215133 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:02.215163 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:02.216944 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:02.217354 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:02.217362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:02.217370 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:02.217533 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:02.217825 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:02.217828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:02.217830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:02.218287 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:02.218487 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:02.218656 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:41:02.218936 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
I20260928 14:41:02.219830 4940 raft_consensus.cc:374] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:02.219872 4940 raft_consensus.cc:729] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88bc85a0bce498b9d10c45227407e5b, State: Initialized, Role: FOLLOWER
I20260928 14:41:02.220292 4940 consensus_queue.cc:260] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88bc85a0bce498b9d10c45227407e5b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43069 } }
I20260928 14:41:02.220434 4940 raft_consensus.cc:388] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:02.220496 4940 raft_consensus.cc:482] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:02.220584 4940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:02.221329 4940 raft_consensus.cc:504] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88bc85a0bce498b9d10c45227407e5b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43069 } }
I20260928 14:41:02.221649 4940 leader_election.cc:302] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c88bc85a0bce498b9d10c45227407e5b; no voters:
I20260928 14:41:02.221940 4940 leader_election.cc:288] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:02.222036 4945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:02.222245 4945 raft_consensus.cc:686] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [term 1 LEADER]: Becoming Leader. State: Replica: c88bc85a0bce498b9d10c45227407e5b, State: Running, Role: LEADER
I20260928 14:41:02.222558 4945 consensus_queue.cc:237] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88bc85a0bce498b9d10c45227407e5b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43069 } }
I20260928 14:41:02.222958 4940 sys_catalog.cc:564] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:02.223948 4947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c88bc85a0bce498b9d10c45227407e5b. Latest consensus state: current_term: 1 leader_uuid: "c88bc85a0bce498b9d10c45227407e5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88bc85a0bce498b9d10c45227407e5b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43069 } } }
I20260928 14:41:02.224048 4947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:02.224319 4946 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c88bc85a0bce498b9d10c45227407e5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c88bc85a0bce498b9d10c45227407e5b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43069 } } }
I20260928 14:41:02.224453 4946 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:02.224505 4954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:02.227298 4954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:02.232409 4954 catalog_manager.cc:1347] Generated new cluster ID: 6796a504b60c4e8188fec4da2c4dbfa1
I20260928 14:41:02.232463 4954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:02.240525 4954 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:02.241528 4954 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:02.247877 4954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c88bc85a0bce498b9d10c45227407e5b: Generated new TSK 0
I20260928 14:41:02.248469 4954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:41:02.278035 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43069
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:02.378520 4966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:02.378710 4966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:02.378749 4966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:02.382326 4966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:02.382366 4966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:02.382426 4966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:02.386952 4966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:02.387773 4966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:02.388620 4966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:41:02.396953 4966 server_base.cc:1034] running on GCE node
W20260928 14:41:02.396926 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:02.396989 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:02.397121 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:02.397496 4966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:02.398130 4966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:02.399292 4966 hybrid_clock.cc:648] HybridClock initialized: now 1790606462399265 us; error 37 us; skew 500 ppm
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:02.401592 4966 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:41:02.402360 4966 webserver.cc:460] Webserver started at http://127.1.201.1:46503/ using document root <none> and password file <none>
I20260928 14:41:02.402877 4966 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:02.402958 4966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:02.403156 4966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:02.404834 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "a2836f7e892b41428e04b14a3cc69ca6"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "0538e3fe6447b8e56ec5e200be69bc9c"
server_key_iv: "ee679fe2d05555714fa7f58be0c3a234"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.405309 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "a2836f7e892b41428e04b14a3cc69ca6"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "0538e3fe6447b8e56ec5e200be69bc9c"
server_key_iv: "ee679fe2d05555714fa7f58be0c3a234"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.408588 4966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:02.410633 4981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:02.411677 4966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:02.411821 4966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "a2836f7e892b41428e04b14a3cc69ca6"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "0538e3fe6447b8e56ec5e200be69bc9c"
server_key_iv: "ee679fe2d05555714fa7f58be0c3a234"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.411929 4966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:02.421139 4966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:02.423807 4966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:02.423957 4966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:02.424407 4966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:02.425072 4966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:02.425132 4966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:02.425189 4966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:02.425231 4966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:02.432883 4966 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:37707
I20260928 14:41:02.432905 5094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:37707 every 8 connection(s)
I20260928 14:41:02.433593 4966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:02.442688 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 4966
I20260928 14:41:02.442809 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:02.443055 1828 external_mini_cluster.cc:1428] Setting key 2f12c9d44e6d92cf44efc82a944396b6
I20260928 14:41:02.445292 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:02.435124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:34283 (local address 127.1.201.62:43069)
0928 14:41:02.435262 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:02.435266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:02.435983 (+ 717us) server_negotiation.cc:407] Connection header received
0928 14:41:02.437164 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:02.437166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:02.437216 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:02.437286 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:02.438327 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.438846 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.439825 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.440044 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.440112 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:02.442056 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:02.442059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:02.442062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:02.442082 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:02.443409 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:02.443932 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:02.443937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:02.443939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:02.444015 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:02.444341 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:02.444343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:02.444344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:02.444492 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:02.444562 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:02.445098 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 14:41:02.445187 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:41:02.446316 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:02.435460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:43069 (local address 127.1.201.1:34283)
0928 14:41:02.435849 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:02.435885 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:02.436993 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:02.437422 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:02.437427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:02.437731 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:02.438207 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:02.438219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.438949 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.438951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:02.439730 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:02.439738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.440149 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.440151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:02.440272 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:02.440655 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:02.440668 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:02.441921 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:02.443513 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:02.443533 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:02.443547 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:02.443834 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:02.444101 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:02.444105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:02.444107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:02.444263 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:02.444590 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:02.444597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:02.444875 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 14:41:02.445066 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 14:41:02.445375 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
I20260928 14:41:02.447654 5095 heartbeater.cc:347] Connected to a master server at 127.1.201.62:43069
I20260928 14:41:02.447860 5095 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:02.448266 5095 heartbeater.cc:511] Master 127.1.201.62:43069 requested a full tablet report, sending...
I20260928 14:41:02.449818 4903 ts_manager.cc:194] Registered new tserver with Master: a2836f7e892b41428e04b14a3cc69ca6 (127.1.201.1:37707)
I20260928 14:41:02.451289 4903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:34283
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:41:02.468458 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43069
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:02.566358 5100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:02.566553 5100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:02.566629 5100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:02.570272 5100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:02.570314 5100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:02.570405 5100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:02.575281 5100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:02.576119 5100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:02.576977 5100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:02.585253 5106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:02.585292 5100 server_base.cc:1034] running on GCE node
W20260928 14:41:02.585260 5105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:02.585261 5108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:02.585928 5100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:02.586522 5100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:02.587672 5100 hybrid_clock.cc:648] HybridClock initialized: now 1790606462587642 us; error 41 us; skew 500 ppm
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:02.590173 5100 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:41:02.590924 5100 webserver.cc:460] Webserver started at http://127.1.201.2:37479/ using document root <none> and password file <none>
I20260928 14:41:02.591428 5100 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:02.591482 5100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:02.591701 5100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:02.593477 5100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "f1827d91deb34db48cce82bcb5b5a9e4"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "c21ad56b5ad136b54586839f6e3048c4"
server_key_iv: "ac51b5f080cf6d404d3d59ebb61363b9"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.594108 5100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "f1827d91deb34db48cce82bcb5b5a9e4"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "c21ad56b5ad136b54586839f6e3048c4"
server_key_iv: "ac51b5f080cf6d404d3d59ebb61363b9"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.597330 5100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 14:41:02.599550 5115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:02.600646 5100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:41:02.600775 5100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "f1827d91deb34db48cce82bcb5b5a9e4"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "c21ad56b5ad136b54586839f6e3048c4"
server_key_iv: "ac51b5f080cf6d404d3d59ebb61363b9"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.600862 5100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:02.623046 5100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:02.625854 5100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:02.626019 5100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:02.626507 5100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:02.627249 5100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:02.627295 5100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:02.627353 5100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:02.627385 5100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:02.635176 5100 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:45335
I20260928 14:41:02.635255 5228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:45335 every 8 connection(s)
I20260928 14:41:02.635974 5100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:02.643628 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 5100
I20260928 14:41:02.643762 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:02.644093 1828 external_mini_cluster.cc:1428] Setting key e830ff4170fb1c9f6faca9b5441a62ee
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:02.648517 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:02.637514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:51929 (local address 127.1.201.62:43069)
0928 14:41:02.637644 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:02.637647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:02.638403 (+ 756us) server_negotiation.cc:407] Connection header received
0928 14:41:02.639692 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:02.639696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:02.639752 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:02.639846 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:02.640870 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.641377 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.642441 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.642656 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.642764 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:02.645120 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:02.645124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:02.645127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:02.645171 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:02.646539 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:02.647105 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:02.647112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:02.647114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:02.647197 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:02.647519 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:02.647524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:02.647526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:02.647700 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:02.647810 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:02.648247 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 14:41:02.648357 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:41:02.649180 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:02.637875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:43069 (local address 127.1.201.2:51929)
0928 14:41:02.638273 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:02.638310 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:02.639487 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:02.640000 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:02.640006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:02.640319 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:02.640734 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:02.640746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.641490 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.641494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:02.642320 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:02.642329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.642794 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.642798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:02.642947 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:02.643369 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:02.643403 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:02.644967 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:02.646670 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:02.646676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:02.646688 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:02.647014 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:02.647287 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:02.647291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:02.647293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:02.647426 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:02.647810 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:02.647816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:02.648029 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 14:41:02.648203 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 14:41:02.648488 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":105,"threads_started":1}
I20260928 14:41:02.650216 5229 heartbeater.cc:347] Connected to a master server at 127.1.201.62:43069
I20260928 14:41:02.650444 5229 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:02.650985 5229 heartbeater.cc:511] Master 127.1.201.62:43069 requested a full tablet report, sending...
I20260928 14:41:02.651993 4903 ts_manager.cc:194] Registered new tserver with Master: f1827d91deb34db48cce82bcb5b5a9e4 (127.1.201.2:45335)
I20260928 14:41:02.652760 4903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:51929
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:41:02.668491 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43069
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:02.769469 5234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:02.769666 5234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:02.769702 5234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:02.773303 5234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:02.773343 5234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:02.773392 5234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:02.778467 5234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:02.779326 5234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:02.780148 5234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:02.788380 5240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:02.788403 5239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:02.788404 5242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:02.788558 5234 server_base.cc:1034] running on GCE node
I20260928 14:41:02.789011 5234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:02.789515 5234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:02.790700 5234 hybrid_clock.cc:648] HybridClock initialized: now 1790606462790693 us; error 28 us; skew 500 ppm
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:02.793049 5234 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:41:02.793768 5234 webserver.cc:460] Webserver started at http://127.1.201.3:36557/ using document root <none> and password file <none>
I20260928 14:41:02.794279 5234 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:02.794354 5234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:02.794577 5234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:02.796361 5234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "7c901d9e0ce047debc84c70293efa06f"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "fa31b2a71ca82f84834d68811dd06b8f"
server_key_iv: "a06c6ba9607dc1c9d8ea41f6385ecf67"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.796926 5234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "7c901d9e0ce047debc84c70293efa06f"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "fa31b2a71ca82f84834d68811dd06b8f"
server_key_iv: "a06c6ba9607dc1c9d8ea41f6385ecf67"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.800259 5234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260928 14:41:02.802376 5249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:02.803493 5234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:41:02.803640 5234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "7c901d9e0ce047debc84c70293efa06f"
format_stamp: "Formatted at 2026-09-28 14:41:02 on dist-test-slave-0n27"
server_key: "fa31b2a71ca82f84834d68811dd06b8f"
server_key_iv: "a06c6ba9607dc1c9d8ea41f6385ecf67"
server_key_version: "encryptionkey@0"
I20260928 14:41:02.803750 5234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:02.818269 5234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:02.820596 5234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:02.820773 5234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:02.821239 5234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:02.821928 5234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:02.821988 5234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:02.822059 5234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:02.822095 5234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:02.830116 5234 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:33637
I20260928 14:41:02.830140 5362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:33637 every 8 connection(s)
I20260928 14:41:02.830833 5234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:41:02.833390 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 5234
I20260928 14:41:02.833503 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:02.833748 1828 external_mini_cluster.cc:1428] Setting key d01b988d368205aea96742ab37fa41a5
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:02.842420 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:02.832366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:45441 (local address 127.1.201.62:43069)
0928 14:41:02.832472 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:02.832476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:02.833104 (+ 628us) server_negotiation.cc:407] Connection header received
0928 14:41:02.834556 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:02.834559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:02.834615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:02.834702 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:02.835699 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.836175 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.837233 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.837449 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.837551 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:02.839617 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:02.839620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:02.839627 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:02.839651 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:02.840691 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:02.841120 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:02.841125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:02.841126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:02.841207 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:02.841492 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:02.841496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:02.841498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:02.841651 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:02.841745 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:02.842174 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 14:41:02.842279 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:41:02.843005 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:02.832616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:43069 (local address 127.1.201.3:45441)
0928 14:41:02.832978 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:02.833012 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:02.834382 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:02.834860 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:02.834866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:02.835146 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:02.835575 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:02.835602 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.836289 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.836292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:02.837110 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:02.837118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.837583 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.837587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:02.837753 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:02.838221 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:02.838234 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:02.839467 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:02.840792 (+ 1325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:02.840797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:02.840808 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:02.841024 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:02.841287 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:02.841291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:02.841292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:02.841409 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:02.841741 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:02.841746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:02.841971 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 14:41:02.842137 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 14:41:02.842376 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
I20260928 14:41:02.843912 5363 heartbeater.cc:347] Connected to a master server at 127.1.201.62:43069
I20260928 14:41:02.844099 5363 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:02.844483 5363 heartbeater.cc:511] Master 127.1.201.62:43069 requested a full tablet report, sending...
I20260928 14:41:02.845327 4903 ts_manager.cc:194] Registered new tserver with Master: 7c901d9e0ce047debc84c70293efa06f (127.1.201.3:33637)
I20260928 14:41:02.845907 4903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:45441
I20260928 14:41:02.847142 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[4859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606462, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:02.869717 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:02.862455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.1.201.62:43069)
0928 14:41:02.862559 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:02.862563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:02.862710 (+ 147us) server_negotiation.cc:407] Connection header received
0928 14:41:02.862789 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:02.862791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:02.862824 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:02.862890 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:02.863493 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.863952 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.864737 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:02.864957 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:02.865055 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:02.867231 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:02.867234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:02.867236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:02.867257 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:02.868330 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:02.868737 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:02.868739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:02.868741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:02.868805 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:02.869024 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:02.869025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:02.869026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:02.869199 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:02.869294 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:02.869486 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 14:41:02.869560 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 14:41:02.870112 4904 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43004
I20260928 14:41:02.870872 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 4966
I20260928 14:41:02.876065 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 5100
I20260928 14:41:02.881119 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 5234
I20260928 14:41:02.885854 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 4872
2026-09-28T14:41:02Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1925 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[5375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[5375](info): set up 2 sockets
Sep 28 14:41:02 dist-test-slave-0n27 krb5kdc[5375](info): commencing operation
krb5kdc: starting...
W20260928 14:41:03.858466 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:41:03 dist-test-slave-0n27 krb5kdc[5375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:41:03.964193 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:39755
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:04.063227 5388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:04.063426 5388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:04.063462 5388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:04.067068 5388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:04.067114 5388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:04.067133 5388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:04.067188 5388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:04.067222 5388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:04.071838 5388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:04.072626 5388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:04.073431 5388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:04.081037 5394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:04.081110 5388 server_base.cc:1034] running on GCE node
W20260928 14:41:04.081111 5396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:04.081037 5393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:04.081866 5388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:04.082804 5388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:04.083979 5388 hybrid_clock.cc:648] HybridClock initialized: now 1790606464083953 us; error 40 us; skew 500 ppm
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:04.086414 5388 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:41:04.087188 5388 webserver.cc:460] Webserver started at http://127.1.201.62:37657/ using document root <none> and password file <none>
I20260928 14:41:04.087683 5388 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:04.087767 5388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:04.087990 5388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:04.089823 5388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "13b1f9edc86d405b9253ba2f264fd11c"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "4dc803f09d1b65e7f2d377eb2b2c6a41"
server_key_iv: "b4a6c2dd12d39399ef795f66c1ce5e8d"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.090342 5388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "13b1f9edc86d405b9253ba2f264fd11c"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "4dc803f09d1b65e7f2d377eb2b2c6a41"
server_key_iv: "b4a6c2dd12d39399ef795f66c1ce5e8d"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.093585 5388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 14:41:04.095671 5403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:04.096716 5388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:04.096855 5388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "13b1f9edc86d405b9253ba2f264fd11c"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "4dc803f09d1b65e7f2d377eb2b2c6a41"
server_key_iv: "b4a6c2dd12d39399ef795f66c1ce5e8d"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.096952 5388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:04.109421 5388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:04.111686 5388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:04.111848 5388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:04.117841 5388 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:39755
I20260928 14:41:04.117869 5455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:39755 every 8 connection(s)
I20260928 14:41:04.118649 5388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:04.121014 5456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:04.125941 5456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c: Bootstrap starting.
I20260928 14:41:04.128094 5456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:04.128444 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 5388
I20260928 14:41:04.128561 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:04.128789 1828 external_mini_cluster.cc:1428] Setting key 67e229dab7314fcdd8f95dc10106406b
I20260928 14:41:04.128816 5456 log.cc:824] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:04.130605 5456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c: No bootstrap required, opened a new log
I20260928 14:41:04.134363 5456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b1f9edc86d405b9253ba2f264fd11c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39755 } }
I20260928 14:41:04.134562 5456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:04.134634 5456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b1f9edc86d405b9253ba2f264fd11c, State: Initialized, Role: FOLLOWER
I20260928 14:41:04.135082 5456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b1f9edc86d405b9253ba2f264fd11c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39755 } }
I20260928 14:41:04.135262 5456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:04.135330 5456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:04.135422 5456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:04.136461 5456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b1f9edc86d405b9253ba2f264fd11c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39755 } }
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:04.136829 5456 leader_election.cc:302] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13b1f9edc86d405b9253ba2f264fd11c; no voters:
I20260928 14:41:04.137152 5456 leader_election.cc:288] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:04.137243 5461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:04.137455 5461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [term 1 LEADER]: Becoming Leader. State: Replica: 13b1f9edc86d405b9253ba2f264fd11c, State: Running, Role: LEADER
I20260928 14:41:04.137828 5461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b1f9edc86d405b9253ba2f264fd11c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39755 } }
I20260928 14:41:04.138428 5456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:04.139338 5462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13b1f9edc86d405b9253ba2f264fd11c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b1f9edc86d405b9253ba2f264fd11c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39755 } } }
I20260928 14:41:04.139446 5462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:04.139827 5463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b1f9edc86d405b9253ba2f264fd11c. Latest consensus state: current_term: 1 leader_uuid: "13b1f9edc86d405b9253ba2f264fd11c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b1f9edc86d405b9253ba2f264fd11c" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39755 } } }
I20260928 14:41:04.139979 5463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:04.140583 5470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:04.142598 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:04.129731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.1.201.62:39755)
0928 14:41:04.130131 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:04.130152 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:04.130190 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:41:04.130777 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:04.130798 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:04.131075 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:04.131329 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:04.132147 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.133028 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.134161 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.134586 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.134731 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:04.137372 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:04.137379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:04.137395 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:04.137423 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:04.139402 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.139912 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.139919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.139925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.140063 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.140392 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.140396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.140398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.140903 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:04.141101 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:04.141511 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 14:41:04.141800 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":102,"threads_started":1}
I20260928 14:41:04.143617 5470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:04.148408 5470 catalog_manager.cc:1347] Generated new cluster ID: bc2d707ee7e9440a99a5c9e2c768d413
I20260928 14:41:04.148473 5470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:04.155285 5470 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:04.156055 5470 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:04.160270 5470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13b1f9edc86d405b9253ba2f264fd11c: Generated new TSK 0
I20260928 14:41:04.160748 5470 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:41:04.188969 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39755
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:04.305964 5482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:04.306162 5482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:04.306206 5482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:04.309970 5482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:04.310009 5482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:04.310057 5482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:04.314538 5482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:04.315305 5482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:04.316138 5482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:04.324586 5490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:04.324602 5488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:04.324755 5487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:04.324625 5482 server_base.cc:1034] running on GCE node
I20260928 14:41:04.325197 5482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:04.325733 5482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:04.326915 5482 hybrid_clock.cc:648] HybridClock initialized: now 1790606464326892 us; error 40 us; skew 500 ppm
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:04.329269 5482 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:41:04.330030 5482 webserver.cc:460] Webserver started at http://127.1.201.1:35657/ using document root <none> and password file <none>
I20260928 14:41:04.330518 5482 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:04.330574 5482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:04.330788 5482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:04.332569 5482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "aab937eb71dd465a876fc0fde68e191e"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "feddf443b8f63630bfcd0cbf8611c68c"
server_key_iv: "a4c2b8396fff2554ee038f4aea1e0b91"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.333066 5482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "aab937eb71dd465a876fc0fde68e191e"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "feddf443b8f63630bfcd0cbf8611c68c"
server_key_iv: "a4c2b8396fff2554ee038f4aea1e0b91"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.336201 5482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:04.338291 5497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:04.339350 5482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:04.339457 5482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "aab937eb71dd465a876fc0fde68e191e"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "feddf443b8f63630bfcd0cbf8611c68c"
server_key_iv: "a4c2b8396fff2554ee038f4aea1e0b91"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.339535 5482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:04.349061 5482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:04.351408 5482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:04.351565 5482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:04.352041 5482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:04.352747 5482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:04.352790 5482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:04.352851 5482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:04.352886 5482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:04.361716 5482 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:39405
I20260928 14:41:04.361740 5610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:39405 every 8 connection(s)
I20260928 14:41:04.362468 5482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:04.364357 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 5482
I20260928 14:41:04.364486 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:04.364804 1828 external_mini_cluster.cc:1428] Setting key d4f7de6992dc1c1a95e72695ac3beca6
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
I20260928 14:41:04.376950 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:04.364084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:36183 (local address 127.1.201.62:39755)
0928 14:41:04.364232 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:04.364238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:04.365040 (+ 802us) server_negotiation.cc:407] Connection header received
0928 14:41:04.366546 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:04.366549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:04.366591 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:04.366659 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:04.368133 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.369025 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.370162 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.370641 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.370769 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:04.373293 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:04.373296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:04.373298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:04.373319 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:04.374713 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.375255 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.375258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.375259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.375326 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.375597 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.375599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.375600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.375766 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:04.375835 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:04.376627 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 14:41:04.376754 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:41:04.377619 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:04.364401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39755 (local address 127.1.201.1:36183)
0928 14:41:04.364893 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:04.364944 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:04.366347 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:04.366847 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:04.366858 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:04.367321 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:04.367948 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:04.367970 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.369150 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.369153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:04.370051 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:04.370060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.370785 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.370787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:04.370911 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:04.371426 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:04.371440 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:04.373118 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:04.374823 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:04.374829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:04.374843 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:04.375174 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:04.375406 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:04.375408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:04.375409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:04.375521 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:04.375913 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:04.375922 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:04.376331 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 14:41:04.376562 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 14:41:04.376891 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":131,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
I20260928 14:41:04.378782 5611 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39755
I20260928 14:41:04.378989 5611 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:04.379392 5611 heartbeater.cc:511] Master 127.1.201.62:39755 requested a full tablet report, sending...
I20260928 14:41:04.380831 5420 ts_manager.cc:194] Registered new tserver with Master: aab937eb71dd465a876fc0fde68e191e (127.1.201.1:39405)
I20260928 14:41:04.382367 5420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:36183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:41:04.392379 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39755
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:04.490253 5616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:04.490463 5616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:04.490499 5616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:04.494159 5616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:04.494201 5616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:04.494251 5616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:04.499069 5616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:04.499907 5616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:04.500835 5616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:04.509315 5622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:04.509296 5621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:04.509330 5616 server_base.cc:1034] running on GCE node
W20260928 14:41:04.509299 5624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:04.509917 5616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:04.510447 5616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:04.511591 5616 hybrid_clock.cc:648] HybridClock initialized: now 1790606464511579 us; error 25 us; skew 500 ppm
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:04.513993 5616 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:41:04.514663 5616 webserver.cc:460] Webserver started at http://127.1.201.2:37691/ using document root <none> and password file <none>
I20260928 14:41:04.515113 5616 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:04.515195 5616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:04.515384 5616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:04.517186 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "30ea364082944f8f93b7736f8df7909f"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "11086308e84b6f332f23abb69c1533ad"
server_key_iv: "2d6845a593b143fc41868a66140cb93b"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.517854 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "30ea364082944f8f93b7736f8df7909f"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "11086308e84b6f332f23abb69c1533ad"
server_key_iv: "2d6845a593b143fc41868a66140cb93b"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.521121 5616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:41:04.523284 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:04.524441 5616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:04.524555 5616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "30ea364082944f8f93b7736f8df7909f"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "11086308e84b6f332f23abb69c1533ad"
server_key_iv: "2d6845a593b143fc41868a66140cb93b"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.524704 5616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:04.540808 5616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:04.544626 5616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:04.544788 5616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:04.545266 5616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:04.546233 5616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:04.546293 5616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:04.546350 5616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:04.546378 5616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:04.556567 5616 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:39803
I20260928 14:41:04.556591 5744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:39803 every 8 connection(s)
I20260928 14:41:04.557497 5616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:04.567309 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 5616
I20260928 14:41:04.567438 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:04.567776 1828 external_mini_cluster.cc:1428] Setting key 3b224922c26145190509819cb63f1987
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:04.574261 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:04.561093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:58675 (local address 127.1.201.62:39755)
0928 14:41:04.561246 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:04.561251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:04.561269 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:41:04.562424 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:04.562428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:04.562486 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:04.562574 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:04.564079 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.564891 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.566694 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.567117 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.567224 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:04.570666 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:04.570669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:04.570689 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:04.570717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:04.572100 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.572621 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.572624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.572626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.572710 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.573102 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.573105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.573106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.573329 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:04.573425 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:04.573980 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 14:41:04.574092 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:41:04.575102 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:04.560001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39755 (local address 127.1.201.2:58675)
0928 14:41:04.560623 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:04.560680 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:04.562229 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:04.562688 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:04.562694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:04.563201 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:04.563924 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:04.563944 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.565025 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.565031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:04.566555 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:04.566568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.567452 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.567458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:04.567679 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:04.568492 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:04.568517 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:04.570499 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:04.572254 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:04.572259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:04.572272 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:04.572518 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:04.572804 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:04.572807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:04.572809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:04.572967 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:04.573432 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:04.573440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:04.573713 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 14:41:04.573922 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 14:41:04.574203 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":144,"threads_started":1}
I20260928 14:41:04.576231 5745 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39755
I20260928 14:41:04.576437 5745 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:04.576881 5745 heartbeater.cc:511] Master 127.1.201.62:39755 requested a full tablet report, sending...
I20260928 14:41:04.577881 5420 ts_manager.cc:194] Registered new tserver with Master: 30ea364082944f8f93b7736f8df7909f (127.1.201.2:39803)
I20260928 14:41:04.578657 5420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:58675
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:41:04.596036 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39755
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:04.698474 5750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:04.698685 5750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:04.698725 5750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:04.702641 5750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:04.702687 5750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:04.702739 5750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:04.707706 5750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:04.708537 5750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:04.709414 5750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:04.718302 5755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:04.718292 5758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:04.718292 5756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:04.718648 5750 server_base.cc:1034] running on GCE node
I20260928 14:41:04.719004 5750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:04.719540 5750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:04.720695 5750 hybrid_clock.cc:648] HybridClock initialized: now 1790606464720669 us; error 38 us; skew 500 ppm
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:04.723222 5750 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:41:04.723968 5750 webserver.cc:460] Webserver started at http://127.1.201.3:35575/ using document root <none> and password file <none>
I20260928 14:41:04.724489 5750 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:04.724545 5750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:04.724776 5750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:04.726699 5750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "f0cc6c3308614e0c90f5249868ad27c6"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "4cd7a4bf5eeeffabe6963599a5b64367"
server_key_iv: "f4468b76059b885a9851defb5de4ae10"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.727241 5750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "f0cc6c3308614e0c90f5249868ad27c6"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "4cd7a4bf5eeeffabe6963599a5b64367"
server_key_iv: "f4468b76059b885a9851defb5de4ae10"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.730792 5750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:41:04.732981 5765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:04.734175 5750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:04.734299 5750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "f0cc6c3308614e0c90f5249868ad27c6"
format_stamp: "Formatted at 2026-09-28 14:41:04 on dist-test-slave-0n27"
server_key: "4cd7a4bf5eeeffabe6963599a5b64367"
server_key_iv: "f4468b76059b885a9851defb5de4ae10"
server_key_version: "encryptionkey@0"
I20260928 14:41:04.734419 5750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:04.762645 5750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:04.765141 5750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:04.765323 5750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:04.765865 5750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:04.766623 5750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:04.766696 5750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:04.766759 5750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:04.766808 5750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:04.775136 5750 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:45911
I20260928 14:41:04.775178 5878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:45911 every 8 connection(s)
I20260928 14:41:04.775907 5750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:41:04.781095 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 5750
I20260928 14:41:04.781205 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:04.781457 1828 external_mini_cluster.cc:1428] Setting key 66fd8e9574c4d581ccbc1fb38f9c694d
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:04.788779 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:04.777445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:36739 (local address 127.1.201.62:39755)
0928 14:41:04.777568 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:04.777572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:04.778343 (+ 771us) server_negotiation.cc:407] Connection header received
0928 14:41:04.779738 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:04.779741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:04.779795 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:04.779876 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:04.780945 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.781631 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.783053 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.783343 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.783471 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:04.785557 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:04.785560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:04.785562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:04.785584 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:04.787048 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.787503 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.787505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.787507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.787577 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.787845 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.787847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.787848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.787999 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:04.788080 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:04.788590 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 14:41:04.788674 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:41:04.789533 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:04.777765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39755 (local address 127.1.201.3:36739)
0928 14:41:04.778220 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:04.778256 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:04.779537 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:04.780037 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:04.780043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:04.780353 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:04.780801 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:04.780813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.781754 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.781762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:04.782928 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:04.782937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.783502 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.783506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:04.783667 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:04.784119 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:04.784133 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:04.785410 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:04.787160 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:04.787164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:04.787174 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:04.787421 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:04.787650 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:04.787652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:04.787653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:04.787772 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:04.788142 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:04.788147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:04.788391 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 14:41:04.788550 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 14:41:04.788757 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260928 14:41:04.790532 5879 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39755
I20260928 14:41:04.790738 5879 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:04.791141 5879 heartbeater.cc:511] Master 127.1.201.62:39755 requested a full tablet report, sending...
I20260928 14:41:04.792038 5420 ts_manager.cc:194] Registered new tserver with Master: f0cc6c3308614e0c90f5249868ad27c6 (127.1.201.3:45911)
I20260928 14:41:04.792574 5420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:36739
I20260928 14:41:04.794893 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:41:04.818224 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:04.810069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.1.201.1:39405)
0928 14:41:04.810388 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:04.810397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:04.810417 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:41:04.810457 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:04.810461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:04.810601 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:04.810705 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:04.811278 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.811943 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.812687 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.813005 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.813080 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:04.815304 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:04.815312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:04.815321 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:04.815342 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:04.816668 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.817047 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.817052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.817053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.817123 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.817334 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.817336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.817337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.817533 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:04.817682 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:04.817870 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 14:41:04.818005 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":126,"threads_started":1}
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:04.827870 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:04.820623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.1.201.62:39755)
0928 14:41:04.820750 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:04.820755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:04.820832 (+ 77us) server_negotiation.cc:407] Connection header received
0928 14:41:04.820955 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:04.820957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:04.820996 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:04.821075 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:04.821590 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.822098 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.822807 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:04.823013 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:04.823111 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:04.825250 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:04.825252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:04.825254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:04.825275 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:04.826461 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.826834 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.826836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.826838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.826905 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:04.827117 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:04.827119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:04.827120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:04.827356 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:04.827452 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:04.827641 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:41:04.827728 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 14:41:04.828238 5420 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42600
I20260928 14:41:04.828759 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 5482
I20260928 14:41:04.834548 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 5616
I20260928 14:41:04.839888 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 5750
I20260928 14:41:04.844460 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 5388
2026-09-28T14:41:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1958 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5891](info): set up 2 sockets
Sep 28 14:41:04 dist-test-slave-0n27 krb5kdc[5891](info): commencing operation
krb5kdc: starting...
W20260928 14:41:05.904045 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 14:41:06.208259 1828 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.304s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 14:41:06.427277 1828 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.219s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 14:41:06.642921 1828 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.216s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/test-user.keytab.
W20260928 14:41:07.178870 1828 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.536s user 0.000s sys 0.001s
Sep 28 14:41:07 dist-test-slave-0n27 krb5kdc[5891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
W20260928 14:41:08.522236 1828 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.201.62: real 1.322s user 0.000s sys 0.003s
I20260928 14:41:08.522838 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:38891
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:08.630012 5904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:08.630252 5904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:08.630326 5904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:08.634320 5904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:08.634394 5904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:08.634433 5904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:08.634487 5904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:08.634526 5904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:08.639564 5904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:38891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:38891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:08.640616 5904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:08.641573 5904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:08.649212 5912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:08.649190 5909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:08.649255 5910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:08.649318 5904 server_base.cc:1034] running on GCE node
I20260928 14:41:08.650063 5904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:08.651109 5904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:08.652273 5904 hybrid_clock.cc:648] HybridClock initialized: now 1790606468652255 us; error 29 us; skew 500 ppm
Sep 28 14:41:08 dist-test-slave-0n27 krb5kdc[5891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606468, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:08.660980 5904 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:41:08.661705 5904 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:08.661818 5904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:08.662057 5904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:08.663976 5904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "daf71dbff3324b51b9bf865354756913"
format_stamp: "Formatted at 2026-09-28 14:41:08 on dist-test-slave-0n27"
server_key: "c253bdea49661eed7a20d0de8879d8ac"
server_key_iv: "1accb0359fd156c6340f03642b6ce6dc"
server_key_version: "encryptionkey@0"
I20260928 14:41:08.664517 5904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "daf71dbff3324b51b9bf865354756913"
format_stamp: "Formatted at 2026-09-28 14:41:08 on dist-test-slave-0n27"
server_key: "c253bdea49661eed7a20d0de8879d8ac"
server_key_iv: "1accb0359fd156c6340f03642b6ce6dc"
server_key_version: "encryptionkey@0"
I20260928 14:41:08.668056 5904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 14:41:08.670261 5918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:08.671401 5904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260928 14:41:08.671540 5904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "daf71dbff3324b51b9bf865354756913"
format_stamp: "Formatted at 2026-09-28 14:41:08 on dist-test-slave-0n27"
server_key: "c253bdea49661eed7a20d0de8879d8ac"
server_key_iv: "1accb0359fd156c6340f03642b6ce6dc"
server_key_version: "encryptionkey@0"
I20260928 14:41:08.671674 5904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:08.687000 5904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:08.689564 5904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:08.689752 5904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:08.696583 5904 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:38891
I20260928 14:41:08.696825 5970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:38891 every 8 connection(s)
I20260928 14:41:08.697188 5904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:08.698756 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 5904
I20260928 14:41:08.698908 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:08.699206 1828 external_mini_cluster.cc:1428] Setting key e87997c0634c34c7500afaf4a253f286
I20260928 14:41:08.700006 5971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:08.706243 5971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913: Bootstrap starting.
Sep 28 14:41:08 dist-test-slave-0n27 krb5kdc[5891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:08.708806 5971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:08.709638 5971 log.cc:824] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:08.711835 5971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913: No bootstrap required, opened a new log
I20260928 14:41:08.712631 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:08.700436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.1.201.62:38891)
0928 14:41:08.700828 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:08.700843 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:08.700876 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:41:08.701457 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:08.701472 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:08.701771 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:08.702101 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:08.702945 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:08.703864 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:08.705053 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:08.705503 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:08.705619 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:08.708197 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:08.708202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:08.708213 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:08.708240 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:08.709948 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:08.710411 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:08.710417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:08.710422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:08.710535 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:08.710815 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:08.710817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:08.710818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:08.711163 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:08.711301 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:08.711627 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 14:41:08.711968 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":129,"threads_started":1}
I20260928 14:41:08.717154 5971 raft_consensus.cc:348] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf71dbff3324b51b9bf865354756913" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38891 } }
I20260928 14:41:08.717433 5971 raft_consensus.cc:374] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:08.717521 5971 raft_consensus.cc:729] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daf71dbff3324b51b9bf865354756913, State: Initialized, Role: FOLLOWER
I20260928 14:41:08.718011 5971 consensus_queue.cc:260] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf71dbff3324b51b9bf865354756913" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38891 } }
I20260928 14:41:08.718156 5971 raft_consensus.cc:388] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:08.718254 5971 raft_consensus.cc:482] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:08.718364 5971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:08.815774 5971 raft_consensus.cc:504] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf71dbff3324b51b9bf865354756913" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38891 } }
I20260928 14:41:08.816325 5971 leader_election.cc:302] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daf71dbff3324b51b9bf865354756913; no voters:
I20260928 14:41:08.816744 5971 leader_election.cc:288] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:08.816881 5976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:08.817162 5976 raft_consensus.cc:686] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [term 1 LEADER]: Becoming Leader. State: Replica: daf71dbff3324b51b9bf865354756913, State: Running, Role: LEADER
I20260928 14:41:08.817543 5976 consensus_queue.cc:237] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf71dbff3324b51b9bf865354756913" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38891 } }
I20260928 14:41:08.818344 5971 sys_catalog.cc:564] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:08.819657 5978 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daf71dbff3324b51b9bf865354756913. Latest consensus state: current_term: 1 leader_uuid: "daf71dbff3324b51b9bf865354756913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf71dbff3324b51b9bf865354756913" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38891 } } }
I20260928 14:41:08.819643 5977 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daf71dbff3324b51b9bf865354756913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf71dbff3324b51b9bf865354756913" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38891 } } }
I20260928 14:41:08.819809 5977 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:08.819808 5978 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:08.820204 5983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:08.823431 5983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:08.828595 5983 catalog_manager.cc:1347] Generated new cluster ID: 027c483110354130af481d721bb2a596
I20260928 14:41:08.828660 5983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:08.839349 5983 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:08.840131 5983 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:08.843765 5983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913: Generated new TSK 0
I20260928 14:41:08.844313 5983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
W20260928 14:41:09.782579 1828 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.201.1: real 0.930s user 0.000s sys 0.004s
I20260928 14:41:09.783131 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38891
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:09.886626 5997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:09.886825 5997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:09.886862 5997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:09.890756 5997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:09.890799 5997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:09.890923 5997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:09.896239 5997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:09.897099 5997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:09.898080 5997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:09.906847 6002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:09.906847 6005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:09.906870 6003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:09.907053 5997 server_base.cc:1034] running on GCE node
I20260928 14:41:09.907445 5997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:09.908020 5997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:09.909175 5997 hybrid_clock.cc:648] HybridClock initialized: now 1790606469909158 us; error 31 us; skew 500 ppm
Sep 28 14:41:09 dist-test-slave-0n27 krb5kdc[5891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606469, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:09.911710 5997 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:41:09.912391 5997 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:09.912478 5997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:09.912731 5997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:09.914645 5997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "2c4458dd05f64fe3826ded61492a9c0c"
format_stamp: "Formatted at 2026-09-28 14:41:09 on dist-test-slave-0n27"
server_key: "60c5947616a672f64e8c17a93880ba46"
server_key_iv: "0c68ea3fdc66d25e13feb0b057db3191"
server_key_version: "encryptionkey@0"
I20260928 14:41:09.915256 5997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "2c4458dd05f64fe3826ded61492a9c0c"
format_stamp: "Formatted at 2026-09-28 14:41:09 on dist-test-slave-0n27"
server_key: "60c5947616a672f64e8c17a93880ba46"
server_key_iv: "0c68ea3fdc66d25e13feb0b057db3191"
server_key_version: "encryptionkey@0"
I20260928 14:41:09.918833 5997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:41:09.921284 6011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:09.922489 5997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:09.922621 5997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "2c4458dd05f64fe3826ded61492a9c0c"
format_stamp: "Formatted at 2026-09-28 14:41:09 on dist-test-slave-0n27"
server_key: "60c5947616a672f64e8c17a93880ba46"
server_key_iv: "0c68ea3fdc66d25e13feb0b057db3191"
server_key_version: "encryptionkey@0"
I20260928 14:41:09.922739 5997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:09.931325 5997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:09.933730 5997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:09.933923 5997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:09.934427 5997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:09.935161 5997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:09.935236 5997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:09.935304 5997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:09.935348 5997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:09.946763 5997 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:33519
I20260928 14:41:09.946784 6124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:33519 every 8 connection(s)
I20260928 14:41:09.947431 5997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:09.949004 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 5997
I20260928 14:41:09.949117 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:09.949453 1828 external_mini_cluster.cc:1428] Setting key 4aefbe5c3c8c58dc64a63d8312aa906c
Sep 28 14:41:09 dist-test-slave-0n27 krb5kdc[5891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606469, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
I20260928 14:41:09.962939 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:09.949179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:51791 (local address 127.1.201.62:38891)
0928 14:41:09.949457 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:09.949461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:09.950597 (+ 1136us) server_negotiation.cc:407] Connection header received
0928 14:41:09.952536 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:09.952538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:09.952580 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:09.952655 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:09.954156 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:09.954725 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:09.956651 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:09.956872 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:09.956951 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:09.959694 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:09.959697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:09.959699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:09.959718 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:09.961092 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:09.961552 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:09.961560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:09.961561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:09.961627 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:09.962001 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:09.962003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:09.962005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:09.962161 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:09.962258 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:09.962674 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 14:41:09.962776 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260928 14:41:09.963599 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:09.949604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38891 (local address 127.1.201.1:51791)
0928 14:41:09.950469 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:09.950506 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:09.952324 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:09.952846 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:09.952858 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:09.953217 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:09.954002 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:09.954024 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:09.954864 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:09.954871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:09.956520 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:09.956537 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:09.957039 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:09.957047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:09.957292 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:09.957990 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:09.958011 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:09.959538 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:09.961207 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:09.961211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:09.961223 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:09.961450 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:09.961710 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:09.961714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:09.961715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:09.961908 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:09.962257 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:09.962263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:09.962465 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 14:41:09.962622 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 14:41:09.962894 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":173,"threads_started":1}
I20260928 14:41:09.964666 6125 heartbeater.cc:347] Connected to a master server at 127.1.201.62:38891
I20260928 14:41:09.964872 6125 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:09.965260 6125 heartbeater.cc:511] Master 127.1.201.62:38891 requested a full tablet report, sending...
I20260928 14:41:09.966735 5935 ts_manager.cc:194] Registered new tserver with Master: 2c4458dd05f64fe3826ded61492a9c0c (127.1.201.1:33519)
I20260928 14:41:09.967922 5935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:51791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
W20260928 14:41:10.156889 1828 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.201.2: real 0.206s user 0.001s sys 0.002s
I20260928 14:41:10.157407 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38891
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:10.256326 6131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:10.256520 6131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:10.256556 6131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:10.260370 6131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:10.260413 6131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:10.260494 6131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:10.265211 6131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:10.266088 6131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:10.266906 6131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:10.274900 6139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:10.274950 6136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:10.274971 6137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:10.274925 6131 server_base.cc:1034] running on GCE node
I20260928 14:41:10.275478 6131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:10.276062 6131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:10.277233 6131 hybrid_clock.cc:648] HybridClock initialized: now 1790606470277216 us; error 33 us; skew 500 ppm
Sep 28 14:41:10 dist-test-slave-0n27 krb5kdc[5891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:10.279614 6131 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:41:10.280248 6131 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:10.280330 6131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:10.280534 6131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:10.282264 6131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "de2749c520514d0594eff83778e3c093"
format_stamp: "Formatted at 2026-09-28 14:41:10 on dist-test-slave-0n27"
server_key: "61adf8ec89045ba7d98a82b0f0f2d2fb"
server_key_iv: "ba2ea8b0fae6834527d0ebcfa16d9f7a"
server_key_version: "encryptionkey@0"
I20260928 14:41:10.282931 6131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "de2749c520514d0594eff83778e3c093"
format_stamp: "Formatted at 2026-09-28 14:41:10 on dist-test-slave-0n27"
server_key: "61adf8ec89045ba7d98a82b0f0f2d2fb"
server_key_iv: "ba2ea8b0fae6834527d0ebcfa16d9f7a"
server_key_version: "encryptionkey@0"
I20260928 14:41:10.286263 6131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:41:10.288367 6145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:10.289462 6131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:10.289587 6131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "de2749c520514d0594eff83778e3c093"
format_stamp: "Formatted at 2026-09-28 14:41:10 on dist-test-slave-0n27"
server_key: "61adf8ec89045ba7d98a82b0f0f2d2fb"
server_key_iv: "ba2ea8b0fae6834527d0ebcfa16d9f7a"
server_key_version: "encryptionkey@0"
I20260928 14:41:10.289687 6131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:10.305583 6131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:10.307999 6131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:10.308193 6131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:10.308691 6131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:10.309394 6131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:10.309458 6131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:10.309517 6131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:10.309561 6131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:10.317293 6131 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:39603
I20260928 14:41:10.317312 6258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:39603 every 8 connection(s)
I20260928 14:41:10.317862 6131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:10.323043 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 6131
I20260928 14:41:10.323184 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:10.323482 1828 external_mini_cluster.cc:1428] Setting key 4b87d2c6a32e718df3a0a89adad8f8d1
Sep 28 14:41:10 dist-test-slave-0n27 krb5kdc[5891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:10.332229 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.319627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:60347 (local address 127.1.201.62:38891)
0928 14:41:10.319792 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:10.319796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:10.320551 (+ 755us) server_negotiation.cc:407] Connection header received
0928 14:41:10.321951 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:10.321954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:10.322008 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:10.322104 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:10.323333 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.323972 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.325370 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.325667 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.325747 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.328530 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:10.328536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:10.328542 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:10.328575 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:10.330266 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:10.330718 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:10.330724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:10.330729 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:10.330819 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:10.331124 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:10.331129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:10.331134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:10.331388 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:10.331491 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:10.331890 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 14:41:10.332025 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 14:41:10.332759 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.319987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38891 (local address 127.1.201.2:60347)
0928 14:41:10.320426 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:10.320463 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:10.321718 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:10.322262 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:10.322268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:10.322558 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:10.323141 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.323162 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.324103 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.324106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.325253 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.325267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.325829 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.325837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.326074 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.326733 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:10.326762 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:10.328359 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:10.330377 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:10.330381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:10.330391 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:10.330620 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:10.330916 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:10.330918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:10.330919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:10.331034 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:10.331483 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:10.331487 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:10.331681 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 14:41:10.331813 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 14:41:10.332014 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
I20260928 14:41:10.333874 6259 heartbeater.cc:347] Connected to a master server at 127.1.201.62:38891
I20260928 14:41:10.334111 6259 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:10.334511 6259 heartbeater.cc:511] Master 127.1.201.62:38891 requested a full tablet report, sending...
I20260928 14:41:10.335469 5935 ts_manager.cc:194] Registered new tserver with Master: de2749c520514d0594eff83778e3c093 (127.1.201.2:39603)
I20260928 14:41:10.335995 5935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:60347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:41:10.350515 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38891
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:10.448793 6264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:10.448976 6264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:10.449046 6264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:10.452577 6264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:10.452618 6264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:10.452689 6264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:10.457116 6264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:10.457969 6264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:10.458804 6264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:10.466485 6270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:10.466511 6269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:10.466620 6264 server_base.cc:1034] running on GCE node
W20260928 14:41:10.466481 6272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:10.467196 6264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:10.467741 6264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:10.468914 6264 hybrid_clock.cc:648] HybridClock initialized: now 1790606470468898 us; error 28 us; skew 500 ppm
Sep 28 14:41:10 dist-test-slave-0n27 krb5kdc[5891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:10.471275 6264 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:41:10.471969 6264 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:10.472051 6264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:10.472234 6264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:10.474011 6264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "624f5e57dfdc405a98a76676777527a0"
format_stamp: "Formatted at 2026-09-28 14:41:10 on dist-test-slave-0n27"
server_key: "fcb16e36aa8f65d5a2e1481dfb4adbbd"
server_key_iv: "0820ecc3464f6d00b71ddb08bef528b2"
server_key_version: "encryptionkey@0"
I20260928 14:41:10.474539 6264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "624f5e57dfdc405a98a76676777527a0"
format_stamp: "Formatted at 2026-09-28 14:41:10 on dist-test-slave-0n27"
server_key: "fcb16e36aa8f65d5a2e1481dfb4adbbd"
server_key_iv: "0820ecc3464f6d00b71ddb08bef528b2"
server_key_version: "encryptionkey@0"
I20260928 14:41:10.478101 6264 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:41:10.480053 6278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:10.481029 6264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:10.481144 6264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "624f5e57dfdc405a98a76676777527a0"
format_stamp: "Formatted at 2026-09-28 14:41:10 on dist-test-slave-0n27"
server_key: "fcb16e36aa8f65d5a2e1481dfb4adbbd"
server_key_iv: "0820ecc3464f6d00b71ddb08bef528b2"
server_key_version: "encryptionkey@0"
I20260928 14:41:10.481242 6264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:10.502425 6264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:10.504971 6264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:10.505134 6264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:10.505594 6264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:10.506269 6264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:10.506331 6264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:10.506387 6264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:10.506428 6264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:10.514022 6264 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:46055
I20260928 14:41:10.514055 6391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:46055 every 8 connection(s)
I20260928 14:41:10.514448 6264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:41:10.515251 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 6264
I20260928 14:41:10.515369 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:10.515651 1828 external_mini_cluster.cc:1428] Setting key d69b441c80a54fff88cb6237d160f197
Sep 28 14:41:10 dist-test-slave-0n27 krb5kdc[5891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:10.525841 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.517077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:49999 (local address 127.1.201.62:38891)
0928 14:41:10.517202 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:10.517206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:10.517219 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:41:10.518202 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:10.518204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:10.518246 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:10.518311 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:10.519299 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.519854 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.520844 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.521039 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.521109 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.523014 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:10.523016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:10.523018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:10.523037 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:10.524325 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:10.524672 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:10.524675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:10.524676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:10.524755 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:10.525002 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:10.525004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:10.525005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:10.525185 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:10.525249 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:10.525616 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 14:41:10.525696 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:41:10.526557 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.516156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38891 (local address 127.1.201.3:49999)
0928 14:41:10.516644 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:10.516692 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:10.518029 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:10.518443 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:10.518449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:10.518836 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:10.519199 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.519210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.519966 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.519968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.520757 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.520765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.521147 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.521149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.521282 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.521653 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:10.521666 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:10.522875 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:10.524403 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:10.524406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:10.524414 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:10.524602 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:10.524824 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:10.524825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:10.524826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:10.524937 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:10.525260 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:10.525264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:10.525449 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 14:41:10.525574 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 14:41:10.525767 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":123,"threads_started":1}
I20260928 14:41:10.527487 6392 heartbeater.cc:347] Connected to a master server at 127.1.201.62:38891
I20260928 14:41:10.527715 6392 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:10.528169 6392 heartbeater.cc:511] Master 127.1.201.62:38891 requested a full tablet report, sending...
I20260928 14:41:10.529124 5934 ts_manager.cc:194] Registered new tserver with Master: 624f5e57dfdc405a98a76676777527a0 (127.1.201.3:46055)
I20260928 14:41:10.529639 5934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:49999
I20260928 14:41:10.540623 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:41:10.548298 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.542326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.1.201.62:38891)
0928 14:41:10.542414 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:10.542417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:10.542607 (+ 190us) server_negotiation.cc:407] Connection header received
0928 14:41:10.542687 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:10.542688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:10.542717 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:10.542774 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:10.543351 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.543758 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.544444 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.544616 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.544698 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.545748 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:10.545751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:10.545752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:10.545770 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:10.547038 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:10.547353 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:10.547355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:10.547356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:10.547453 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:10.547665 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:10.547667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:10.547668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:10.547840 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:10.547934 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:10.548116 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 14:41:10.548203 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:41:10.550652 5934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:41:10.553252 5934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:41:10.566780 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.560814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39603 (local address 127.0.0.1:59822)
0928 14:41:10.561611 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:10.561623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:10.561711 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:10.562325 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:10.562330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:10.562349 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:10.562463 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.562469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.563845 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.563849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.565396 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.565405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.566351 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.566354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.566459 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.566482 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:41:10.566551 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:41:10.566615 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":110,"thread_start_us":60,"threads_started":1}
I20260928 14:41:10.566943 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.560978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.1.201.2:39603)
0928 14:41:10.561972 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:10.561989 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:10.562005 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:41:10.562059 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:10.562063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:10.562192 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:10.562294 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:10.562576 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.563649 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.565519 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.566196 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.566326 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.566471 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:10.566708 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 14:41:10.566806 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":124,"threads_started":1}
I20260928 14:41:10.567047 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.560653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:33519 (local address 127.0.0.1:35012)
0928 14:41:10.561539 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:10.561579 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:10.561702 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:10.562465 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:10.562467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:10.562480 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:10.562571 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.562576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.563819 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.563824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.565583 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.565590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.566561 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.566565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.566645 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.566685 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 14:41:10.566763 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:41:10.566927 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":104,"threads_started":1}
I20260928 14:41:10.567241 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.561290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.1.201.1:33519)
0928 14:41:10.561972 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:10.561978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:10.562003 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:41:10.562057 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:10.562060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:10.562198 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:10.562338 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:10.562678 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.563591 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.565693 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.566463 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.566568 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.566699 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:10.566785 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:41:10.566962 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":103,"threads_started":1}
I20260928 14:41:10.567346 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.561341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.1.201.3:46055)
0928 14:41:10.562491 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:10.562498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:10.562512 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:41:10.562589 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:10.562594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:10.562748 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:10.562879 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:10.563079 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.564185 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.565378 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.566009 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.566107 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.566231 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:10.567084 (+ 853us) server_negotiation.cc:299] Negotiation successful
0928 14:41:10.567197 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1082,"thread_start_us":101,"threads_started":1}
I20260928 14:41:10.566633 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.561048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:46055 (local address 127.0.0.1:40888)
0928 14:41:10.561792 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:10.561804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:10.561870 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:10.562882 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:10.562885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:10.562902 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:10.562974 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.562979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.564343 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.564346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.565266 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.565275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.566122 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.566125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.566241 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.566292 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 14:41:10.566380 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:41:10.566504 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260928 14:41:10.569818 6326 tablet_service.cc:1467] Processing CreateTablet for tablet 27126ef7d9b8463f9553d29d5b74c9cb (DEFAULT_TABLE table=test-table [id=aca96f7770e345f7be6bb82eeb624993]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:10.569867 6059 tablet_service.cc:1467] Processing CreateTablet for tablet 27126ef7d9b8463f9553d29d5b74c9cb (DEFAULT_TABLE table=test-table [id=aca96f7770e345f7be6bb82eeb624993]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:10.569867 6193 tablet_service.cc:1467] Processing CreateTablet for tablet 27126ef7d9b8463f9553d29d5b74c9cb (DEFAULT_TABLE table=test-table [id=aca96f7770e345f7be6bb82eeb624993]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:10.570722 6326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27126ef7d9b8463f9553d29d5b74c9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:10.570742 6193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27126ef7d9b8463f9553d29d5b74c9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:10.570742 6059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27126ef7d9b8463f9553d29d5b74c9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:10.576078 6410 tablet_bootstrap.cc:492] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: Bootstrap starting.
I20260928 14:41:10.577342 6412 tablet_bootstrap.cc:492] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: Bootstrap starting.
I20260928 14:41:10.577688 6411 tablet_bootstrap.cc:492] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c: Bootstrap starting.
I20260928 14:41:10.578353 6410 tablet_bootstrap.cc:654] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:10.579331 6410 log.cc:824] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:10.579579 6411 tablet_bootstrap.cc:654] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:10.580251 6411 log.cc:824] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:10.581036 6410 tablet_bootstrap.cc:492] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: No bootstrap required, opened a new log
I20260928 14:41:10.581212 6410 ts_tablet_manager.cc:1397] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:41:10.582094 6411 tablet_bootstrap.cc:492] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c: No bootstrap required, opened a new log
I20260928 14:41:10.582273 6411 ts_tablet_manager.cc:1397] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 14:41:10.582353 6412 tablet_bootstrap.cc:654] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:10.583292 6412 log.cc:824] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:10.584928 6410 raft_consensus.cc:348] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4458dd05f64fe3826ded61492a9c0c" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33519 } } peers { permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 } } peers { permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 } }
I20260928 14:41:10.585191 6410 raft_consensus.cc:374] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:10.585247 6410 raft_consensus.cc:729] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 624f5e57dfdc405a98a76676777527a0, State: Initialized, Role: FOLLOWER
I20260928 14:41:10.585666 6410 consensus_queue.cc:260] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4458dd05f64fe3826ded61492a9c0c" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33519 } } peers { permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 } } peers { permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 } }
I20260928 14:41:10.586290 6392 heartbeater.cc:503] Master 127.1.201.62:38891 was elected leader, sending a full tablet report...
I20260928 14:41:10.586576 6410 ts_tablet_manager.cc:1428] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:41:10.586509 6411 raft_consensus.cc:348] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4458dd05f64fe3826ded61492a9c0c" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33519 } } peers { permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 } } peers { permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 } }
I20260928 14:41:10.586731 6411 raft_consensus.cc:374] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:10.586794 6411 raft_consensus.cc:729] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4458dd05f64fe3826ded61492a9c0c, State: Initialized, Role: FOLLOWER
I20260928 14:41:10.587158 6411 consensus_queue.cc:260] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4458dd05f64fe3826ded61492a9c0c" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33519 } } peers { permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 } } peers { permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 } }
I20260928 14:41:10.587795 6125 heartbeater.cc:503] Master 127.1.201.62:38891 was elected leader, sending a full tablet report...
I20260928 14:41:10.587842 6412 tablet_bootstrap.cc:492] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: No bootstrap required, opened a new log
I20260928 14:41:10.587983 6411 ts_tablet_manager.cc:1428] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260928 14:41:10.588052 6412 ts_tablet_manager.cc:1397] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260928 14:41:10.590883 6412 raft_consensus.cc:348] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4458dd05f64fe3826ded61492a9c0c" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33519 } } peers { permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 } } peers { permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 } }
I20260928 14:41:10.591140 6412 raft_consensus.cc:374] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:10.591194 6412 raft_consensus.cc:729] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2749c520514d0594eff83778e3c093, State: Initialized, Role: FOLLOWER
I20260928 14:41:10.591537 6412 consensus_queue.cc:260] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4458dd05f64fe3826ded61492a9c0c" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33519 } } peers { permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 } } peers { permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 } }
I20260928 14:41:10.592249 6412 ts_tablet_manager.cc:1428] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260928 14:41:10.592594 6259 heartbeater.cc:503] Master 127.1.201.62:38891 was elected leader, sending a full tablet report...
I20260928 14:41:10.691726 6417 raft_consensus.cc:482] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:41:10.691890 6417 raft_consensus.cc:504] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4458dd05f64fe3826ded61492a9c0c" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33519 } } peers { permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 } } peers { permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 } }
I20260928 14:41:10.693042 6417 leader_election.cc:288] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2749c520514d0594eff83778e3c093 (127.1.201.2:39603), 624f5e57dfdc405a98a76676777527a0 (127.1.201.3:46055)
I20260928 14:41:10.697916 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.693400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:39071 (local address 127.1.201.3:46055)
0928 14:41:10.693525 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:10.693531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:10.693846 (+ 315us) server_negotiation.cc:407] Connection header received
0928 14:41:10.694177 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:10.694190 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:10.694228 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:10.694310 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:10.694524 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.695203 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.696748 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.697348 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.697470 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.697535 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:10.697716 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 14:41:10.697806 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:41:10.697945 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.693265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:46055 (local address 127.1.201.1:39071)
0928 14:41:10.693752 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:10.693763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:10.693859 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:10.694324 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:10.694329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:10.694347 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:10.694428 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.694432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.695297 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.695299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.696632 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.696644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.697487 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.697490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.697571 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.697591 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:41:10.697696 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:41:10.697769 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":75,"thread_start_us":117,"threads_started":1}
W20260928 14:41:10.698616 6126 tablet.cc:2369] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:41:10.698666 6346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27126ef7d9b8463f9553d29d5b74c9cb" candidate_uuid: "2c4458dd05f64fe3826ded61492a9c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "624f5e57dfdc405a98a76676777527a0" is_pre_election: true
I20260928 14:41:10.698973 6346 raft_consensus.cc:2454] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4458dd05f64fe3826ded61492a9c0c in term 0.
I20260928 14:41:10.699394 6012 leader_election.cc:302] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4458dd05f64fe3826ded61492a9c0c, 624f5e57dfdc405a98a76676777527a0; no voters:
I20260928 14:41:10.699502 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.693314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:34843 (local address 127.1.201.2:39603)
0928 14:41:10.693458 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:10.693462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:10.693756 (+ 294us) server_negotiation.cc:407] Connection header received
0928 14:41:10.693958 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:10.693960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:10.694005 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:10.694071 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:10.694332 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.695095 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.697587 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.698232 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.698328 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.698379 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:10.699322 (+ 943us) server_negotiation.cc:299] Negotiation successful
0928 14:41:10.699397 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:41:10.699555 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.693240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39603 (local address 127.1.201.1:34843)
0928 14:41:10.693657 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:10.693671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:10.693796 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:10.694094 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:10.694096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:10.694110 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:10.694220 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.694224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.695193 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.695195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.696644 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:10.696651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.698354 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.698358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:10.699181 (+ 823us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.699211 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:41:10.699286 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:41:10.699366 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":74,"thread_start_us":121,"threads_started":1}
I20260928 14:41:10.699708 6417 raft_consensus.cc:2781] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:41:10.699783 6417 raft_consensus.cc:482] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:41:10.699819 6417 raft_consensus.cc:3037] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:10.700155 6213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27126ef7d9b8463f9553d29d5b74c9cb" candidate_uuid: "2c4458dd05f64fe3826ded61492a9c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2749c520514d0594eff83778e3c093" is_pre_election: true
I20260928 14:41:10.700456 6213 raft_consensus.cc:2454] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4458dd05f64fe3826ded61492a9c0c in term 0.
I20260928 14:41:10.701038 6417 raft_consensus.cc:504] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4458dd05f64fe3826ded61492a9c0c" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33519 } } peers { permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 } } peers { permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 } }
I20260928 14:41:10.701380 6417 leader_election.cc:288] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [CANDIDATE]: Term 1 election: Requested vote from peers de2749c520514d0594eff83778e3c093 (127.1.201.2:39603), 624f5e57dfdc405a98a76676777527a0 (127.1.201.3:46055)
I20260928 14:41:10.701751 6346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27126ef7d9b8463f9553d29d5b74c9cb" candidate_uuid: "2c4458dd05f64fe3826ded61492a9c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "624f5e57dfdc405a98a76676777527a0"
I20260928 14:41:10.701887 6346 raft_consensus.cc:3037] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:10.703027 6346 raft_consensus.cc:2454] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4458dd05f64fe3826ded61492a9c0c in term 1.
I20260928 14:41:10.703259 6213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27126ef7d9b8463f9553d29d5b74c9cb" candidate_uuid: "2c4458dd05f64fe3826ded61492a9c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2749c520514d0594eff83778e3c093"
I20260928 14:41:10.703312 6012 leader_election.cc:302] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4458dd05f64fe3826ded61492a9c0c, 624f5e57dfdc405a98a76676777527a0; no voters:
I20260928 14:41:10.703365 6213 raft_consensus.cc:3037] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:10.704516 6213 raft_consensus.cc:2454] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4458dd05f64fe3826ded61492a9c0c in term 1.
I20260928 14:41:10.704710 6417 raft_consensus.cc:2781] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:10.704900 6417 raft_consensus.cc:686] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 1 LEADER]: Becoming Leader. State: Replica: 2c4458dd05f64fe3826ded61492a9c0c, State: Running, Role: LEADER
I20260928 14:41:10.705214 6417 consensus_queue.cc:237] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4458dd05f64fe3826ded61492a9c0c" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33519 } } peers { permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 } } peers { permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 } }
I20260928 14:41:10.708364 5935 catalog_manager.cc:5495] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4458dd05f64fe3826ded61492a9c0c (127.1.201.1). New cstate: current_term: 1 leader_uuid: "2c4458dd05f64fe3826ded61492a9c0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4458dd05f64fe3826ded61492a9c0c" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 } health_report { overall_health: UNKNOWN } } }
I20260928 14:41:10.753095 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.749962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.1.201.1:33519)
0928 14:41:10.750100 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:10.750103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:10.750115 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:10.750210 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:10.750212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:10.750264 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:10.750349 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:10.750584 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.751050 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.751860 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:10.752082 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:10.752156 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:10.752417 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:10.752806 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:10.752890 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:10.752947 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 14:41:10.752994 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:41:10.758559 6213 raft_consensus.cc:1261] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093 [term 1 FOLLOWER]: Refusing update from remote peer 2c4458dd05f64fe3826ded61492a9c0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:10.758559 6346 raft_consensus.cc:1261] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0 [term 1 FOLLOWER]: Refusing update from remote peer 2c4458dd05f64fe3826ded61492a9c0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:10.759238 6421 consensus_queue.cc:1035] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2749c520514d0594eff83778e3c093" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:10.759378 6417 consensus_queue.cc:1035] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "624f5e57dfdc405a98a76676777527a0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:41:10.765146 6393 tablet.cc:2369] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:41:10.767969 6426 mvcc.cc:205] Tried to move back new op lower bound from 7334324104220262400 to 7334324104011046912. Current Snapshot: MvccSnapshot[applied={T|T < 7334324104220262400}]
I20260928 14:41:10.769547 6425 mvcc.cc:205] Tried to move back new op lower bound from 7334324104220262400 to 7334324104011046912. Current Snapshot: MvccSnapshot[applied={T|T < 7334324104220262400}]
I20260928 14:41:10.773396 5935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:10.773505 5935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:10.775745 5935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P daf71dbff3324b51b9bf865354756913: Sending DeleteTablet for 3 replicas of tablet 27126ef7d9b8463f9553d29d5b74c9cb
I20260928 14:41:10.776407 6059 tablet_service.cc:1514] Processing DeleteTablet for tablet 27126ef7d9b8463f9553d29d5b74c9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:10 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:35012
I20260928 14:41:10.776407 6193 tablet_service.cc:1514] Processing DeleteTablet for tablet 27126ef7d9b8463f9553d29d5b74c9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:10 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:59822
I20260928 14:41:10.776535 6326 tablet_service.cc:1514] Processing DeleteTablet for tablet 27126ef7d9b8463f9553d29d5b74c9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:10 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:40888
I20260928 14:41:10.776827 6435 tablet_replica.cc:330] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c: stopping tablet replica
I20260928 14:41:10.776957 6434 tablet_replica.cc:330] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: stopping tablet replica
I20260928 14:41:10.777091 6435 raft_consensus.cc:2229] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:41:10.777204 6434 raft_consensus.cc:2229] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:10.777376 6435 raft_consensus.cc:2258] T 27126ef7d9b8463f9553d29d5b74c9cb P 2c4458dd05f64fe3826ded61492a9c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:10.777472 6434 raft_consensus.cc:2258] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:10.777657 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 5997
I20260928 14:41:10.779553 6436 tablet_replica.cc:330] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: stopping tablet replica
I20260928 14:41:10.779834 6436 raft_consensus.cc:2229] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:10.780084 6436 raft_consensus.cc:2258] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:10.780429 6434 ts_tablet_manager.cc:1905] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:10.781304 6436 ts_tablet_manager.cc:1905] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:10.782581 6434 ts_tablet_manager.cc:1918] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:10.782692 6434 log.cc:1196] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/27126ef7d9b8463f9553d29d5b74c9cb
I20260928 14:41:10.782980 6434 ts_tablet_manager.cc:1939] T 27126ef7d9b8463f9553d29d5b74c9cb P de2749c520514d0594eff83778e3c093: Deleting consensus metadata
I20260928 14:41:10.784106 5921 catalog_manager.cc:4841] TS de2749c520514d0594eff83778e3c093 (127.1.201.2:39603): tablet 27126ef7d9b8463f9553d29d5b74c9cb (table test-table [id=aca96f7770e345f7be6bb82eeb624993]) successfully deleted
I20260928 14:41:10.784624 6436 ts_tablet_manager.cc:1918] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:10.784726 6436 log.cc:1196] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/27126ef7d9b8463f9553d29d5b74c9cb
I20260928 14:41:10.785035 6436 ts_tablet_manager.cc:1939] T 27126ef7d9b8463f9553d29d5b74c9cb P 624f5e57dfdc405a98a76676777527a0: Deleting consensus metadata
W20260928 14:41:10.785576 5922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:33519 (error 108)
I20260928 14:41:10.786119 5919 catalog_manager.cc:4841] TS 624f5e57dfdc405a98a76676777527a0 (127.1.201.3:46055): tablet 27126ef7d9b8463f9553d29d5b74c9cb (table test-table [id=aca96f7770e345f7be6bb82eeb624993]) successfully deleted
I20260928 14:41:10.786361 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 6131
I20260928 14:41:10.786548 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.786270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:33519 (local address 127.0.0.1:35034)
0928 14:41:10.786392 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:10.786464 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:33519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 14:41:10.787761 5922 catalog_manager.cc:4584] TS 2c4458dd05f64fe3826ded61492a9c0c (127.1.201.1:33519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27126ef7d9b8463f9553d29d5b74c9cb: Network error: Client connection negotiation failed: client connection to 127.1.201.1:33519: connect: Connection refused (error 111)
I20260928 14:41:10.791967 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 6264
I20260928 14:41:10.797220 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 5904
2026-09-28T14:41:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5952 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T14:41:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:10Z Disabled control of system clock
I20260928 14:41:10.825366 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:43833
--webserver_interface=127.1.201.62
--webserver_port=0
--builtin_ntp_servers=127.1.201.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:43833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:41:10.925428 6441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:10.925652 6441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:10.925681 6441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:10.929575 6441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:10.929629 6441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:10.929644 6441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:10.929693 6441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:10.929721 6441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:10.934015 6441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:43833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:43833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:10.934810 6441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:10.935925 6441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:10.943413 6446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:10.943420 6447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:10.943430 6449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:10.943626 6441 server_base.cc:1034] running on GCE node
I20260928 14:41:10.944303 6441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:10.945262 6441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:10.946436 6441 hybrid_clock.cc:648] HybridClock initialized: now 1790606470946423 us; error 22 us; skew 500 ppm
I20260928 14:41:10.948050 6441 webserver.cc:460] Webserver started at http://127.1.201.62:44851/ using document root <none> and password file <none>
I20260928 14:41:10.948565 6441 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:10.948652 6441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:10.948859 6441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:10.950757 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "134145db8cbb420790975ab1751a77c4"
format_stamp: "Formatted at 2026-09-28 14:41:10 on dist-test-slave-0n27"
server_key: "755e2a4bafd71221affbe224ac4ce027"
server_key_iv: "d49e849d9883158bd90136b16cad422a"
server_key_version: "encryptionkey@0"
I20260928 14:41:10.951299 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "134145db8cbb420790975ab1751a77c4"
format_stamp: "Formatted at 2026-09-28 14:41:10 on dist-test-slave-0n27"
server_key: "755e2a4bafd71221affbe224ac4ce027"
server_key_iv: "d49e849d9883158bd90136b16cad422a"
server_key_version: "encryptionkey@0"
I20260928 14:41:10.954587 6441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:41:10.956624 6455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:10.957679 6441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:10.957831 6441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "134145db8cbb420790975ab1751a77c4"
format_stamp: "Formatted at 2026-09-28 14:41:10 on dist-test-slave-0n27"
server_key: "755e2a4bafd71221affbe224ac4ce027"
server_key_iv: "d49e849d9883158bd90136b16cad422a"
server_key_version: "encryptionkey@0"
I20260928 14:41:10.957940 6441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:10.961896 6441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:10.962391 6441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:10.962555 6441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:10.969008 6441 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:43833
I20260928 14:41:10.969012 6507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:43833 every 8 connection(s)
I20260928 14:41:10.969758 6441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:10.969936 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 6441
I20260928 14:41:10.970050 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:10.970319 1828 external_mini_cluster.cc:1428] Setting key 5f74006185fd380b85d1c80e8666ca0d
I20260928 14:41:10.972563 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:10.975482 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:10.971498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.1.201.62:43833)
0928 14:41:10.972502 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:10.972513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:10.972542 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:41:10.973109 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:10.973122 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:10.973414 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:10.973683 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:10.974369 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:10.974374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:10.974414 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:10.974423 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:10.974469 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:10.974483 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:10.974624 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:10.974692 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:41:10.974911 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":579,"threads_started":1}
I20260928 14:41:10.978076 6508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4: Bootstrap starting.
I20260928 14:41:10.980147 6508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:10.980823 6508 log.cc:824] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:10.982623 6508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4: No bootstrap required, opened a new log
I20260928 14:41:10.985338 6508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134145db8cbb420790975ab1751a77c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43833 } }
I20260928 14:41:10.985502 6508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:10.985577 6508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134145db8cbb420790975ab1751a77c4, State: Initialized, Role: FOLLOWER
I20260928 14:41:10.986016 6508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134145db8cbb420790975ab1751a77c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43833 } }
I20260928 14:41:10.986150 6508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:10.986209 6508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:10.986305 6508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:10.987025 6508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134145db8cbb420790975ab1751a77c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43833 } }
I20260928 14:41:10.987318 6508 leader_election.cc:302] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 134145db8cbb420790975ab1751a77c4; no voters:
I20260928 14:41:10.987576 6508 leader_election.cc:288] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:10.987720 6513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:10.987962 6513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [term 1 LEADER]: Becoming Leader. State: Replica: 134145db8cbb420790975ab1751a77c4, State: Running, Role: LEADER
I20260928 14:41:10.988242 6513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134145db8cbb420790975ab1751a77c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43833 } }
I20260928 14:41:10.988679 6508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:10.989634 6515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 134145db8cbb420790975ab1751a77c4. Latest consensus state: current_term: 1 leader_uuid: "134145db8cbb420790975ab1751a77c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134145db8cbb420790975ab1751a77c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43833 } } }
I20260928 14:41:10.989740 6515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:10.989892 6514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "134145db8cbb420790975ab1751a77c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134145db8cbb420790975ab1751a77c4" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43833 } } }
I20260928 14:41:10.989970 6514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:10.990823 6522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:10.993194 6522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:10.998095 6522 catalog_manager.cc:1347] Generated new cluster ID: 8ce998675297405d8cbad0b3e1cd08e8
I20260928 14:41:10.998173 6522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:11.012743 6522 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:11.012998 6522 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:11.021528 6522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4: Generated new TSK 0
I20260928 14:41:11.022178 6522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:41:11.033861 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43833
--builtin_ntp_servers=127.1.201.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:41:11.131450 6532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:11.131701 6532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:11.131769 6532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:11.135327 6532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:11.135388 6532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:11.135466 6532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:11.139819 6532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:11.140671 6532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:11.141711 6532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:11.150005 6538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:11.150064 6532 server_base.cc:1034] running on GCE node
W20260928 14:41:11.149993 6537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:11.150009 6540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:11.150504 6532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:11.151026 6532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:11.152196 6532 hybrid_clock.cc:648] HybridClock initialized: now 1790606471152184 us; error 33 us; skew 500 ppm
I20260928 14:41:11.153685 6532 webserver.cc:460] Webserver started at http://127.1.201.1:41577/ using document root <none> and password file <none>
I20260928 14:41:11.154197 6532 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:11.154277 6532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:11.154474 6532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:11.156168 6532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "240b6f7c9e5c4f3cb9cfa14850dea151"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "dd380a173c16e0baaa143c005121b05b"
server_key_iv: "c3d4d923278880ffffafdddb81c774f1"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.156668 6532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "240b6f7c9e5c4f3cb9cfa14850dea151"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "dd380a173c16e0baaa143c005121b05b"
server_key_iv: "c3d4d923278880ffffafdddb81c774f1"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.159895 6532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:11.162137 6546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:11.163156 6532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:11.163277 6532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "240b6f7c9e5c4f3cb9cfa14850dea151"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "dd380a173c16e0baaa143c005121b05b"
server_key_iv: "c3d4d923278880ffffafdddb81c774f1"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.163374 6532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:11.167441 6532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:11.167959 6532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:11.168140 6532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:11.168610 6532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:11.169391 6532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:11.169457 6532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:11.169533 6532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:11.169579 6532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:11.177492 6532 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:38919
I20260928 14:41:11.177520 6659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:38919 every 8 connection(s)
I20260928 14:41:11.178573 6532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:11.179100 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 6532
I20260928 14:41:11.179226 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:11.179474 1828 external_mini_cluster.cc:1428] Setting key f712203d163cca90803e162a7b0b9a71
I20260928 14:41:11.181443 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43833
--builtin_ntp_servers=127.1.201.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 14:41:11.184525 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.180205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:41489 (local address 127.1.201.62:43833)
0928 14:41:11.180331 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.180336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.181414 (+ 1078us) server_negotiation.cc:407] Connection header received
0928 14:41:11.182966 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.182972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.183020 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.183103 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.184068 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.184074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.184079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.184094 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.184116 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.184126 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.184218 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.184285 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.184336 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:41:11.185178 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.180591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:43833 (local address 127.1.201.1:41489)
0928 14:41:11.181267 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.181312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:11.182729 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:11.183221 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:11.183227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:11.183304 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:11.183840 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:11.183841 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:11.183875 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:11.183934 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:11.184202 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:11.184205 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:11.184209 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:41:11.184322 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:41:11.184503 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":136,"threads_started":1}
I20260928 14:41:11.186335 6660 heartbeater.cc:347] Connected to a master server at 127.1.201.62:43833
I20260928 14:41:11.186532 6660 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:11.186872 6660 heartbeater.cc:511] Master 127.1.201.62:43833 requested a full tablet report, sending...
I20260928 14:41:11.188248 6472 ts_manager.cc:194] Registered new tserver with Master: 240b6f7c9e5c4f3cb9cfa14850dea151 (127.1.201.1:38919)
W20260928 14:41:11.282068 6663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:11.282248 6663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:11.282279 6663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:11.285666 6663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:11.285703 6663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:11.285749 6663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:11.290534 6663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:11.291361 6663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:11.292510 6663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:11.301095 6671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:11.301223 6669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:11.301107 6668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:11.301281 6663 server_base.cc:1034] running on GCE node
I20260928 14:41:11.301712 6663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:11.302310 6663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:11.303467 6663 hybrid_clock.cc:648] HybridClock initialized: now 1790606471303454 us; error 23 us; skew 500 ppm
I20260928 14:41:11.305071 6663 webserver.cc:460] Webserver started at http://127.1.201.2:38973/ using document root <none> and password file <none>
I20260928 14:41:11.305562 6663 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:11.305645 6663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:11.305888 6663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:11.307796 6663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "3cd86cc2b17d491886c1965f7d2fa611"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "24968b6a8e4596bd49dbab875534ed2f"
server_key_iv: "44906f07e2717cbbc1d79a3d4447ae23"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.308310 6663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "3cd86cc2b17d491886c1965f7d2fa611"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "24968b6a8e4596bd49dbab875534ed2f"
server_key_iv: "44906f07e2717cbbc1d79a3d4447ae23"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.311667 6663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:11.313726 6677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:11.314893 6663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:11.315022 6663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "3cd86cc2b17d491886c1965f7d2fa611"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "24968b6a8e4596bd49dbab875534ed2f"
server_key_iv: "44906f07e2717cbbc1d79a3d4447ae23"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.315137 6663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:11.319407 6663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:11.319921 6663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:11.320102 6663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:11.320582 6663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:11.321376 6663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:11.321446 6663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:11.321516 6663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:11.321570 6663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:11.329703 6663 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:46267
I20260928 14:41:11.329742 6790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:46267 every 8 connection(s)
I20260928 14:41:11.330418 6663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:11.335477 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.331952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:50259 (local address 127.1.201.62:43833)
0928 14:41:11.332116 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.332122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.333011 (+ 889us) server_negotiation.cc:407] Connection header received
0928 14:41:11.334136 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.334138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.334174 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.334261 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.335125 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.335128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.335129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.335140 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.335158 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.335165 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.335234 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.335343 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.335376 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:41:11.336126 6793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.332290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:43833 (local address 127.1.201.2:50259)
0928 14:41:11.332861 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.332898 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:11.333973 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:11.334400 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:11.334406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:11.334488 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:11.334914 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:11.334916 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:11.334946 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:11.334999 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:11.335236 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:11.335241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:11.335245 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:41:11.335353 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:41:11.335508 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
I20260928 14:41:11.337165 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 6663
I20260928 14:41:11.337280 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:11.337256 6791 heartbeater.cc:347] Connected to a master server at 127.1.201.62:43833
I20260928 14:41:11.337455 6791 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:11.337586 1828 external_mini_cluster.cc:1428] Setting key 0ebca140a46fbc9763f181ad7f1ec705
I20260928 14:41:11.337922 6791 heartbeater.cc:511] Master 127.1.201.62:43833 requested a full tablet report, sending...
I20260928 14:41:11.338914 6472 ts_manager.cc:194] Registered new tserver with Master: 3cd86cc2b17d491886c1965f7d2fa611 (127.1.201.2:46267)
I20260928 14:41:11.339521 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43833
--builtin_ntp_servers=127.1.201.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:41:11.439507 6794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:11.439692 6794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:11.439723 6794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:11.443171 6794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:11.443207 6794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:11.443252 6794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:11.447135 6794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:43833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:11.447878 6794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:11.449012 6794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:11.456890 6799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:11.456897 6802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:11.456897 6800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:11.457093 6794 server_base.cc:1034] running on GCE node
I20260928 14:41:11.457508 6794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:11.458189 6794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:11.459338 6794 hybrid_clock.cc:648] HybridClock initialized: now 1790606471459327 us; error 21 us; skew 500 ppm
I20260928 14:41:11.460968 6794 webserver.cc:460] Webserver started at http://127.1.201.3:38619/ using document root <none> and password file <none>
I20260928 14:41:11.461517 6794 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:11.461573 6794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:11.461820 6794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:11.463532 6794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "6f87318b281943bead2c0371c0619f9c"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "95ce787758b1a7037b7049b07d610214"
server_key_iv: "1cfe9ea1e53c6d2dfc8ca75a6d593319"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.464037 6794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "6f87318b281943bead2c0371c0619f9c"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "95ce787758b1a7037b7049b07d610214"
server_key_iv: "1cfe9ea1e53c6d2dfc8ca75a6d593319"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.467492 6794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:11.469553 6808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:11.470644 6794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:11.470757 6794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "6f87318b281943bead2c0371c0619f9c"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "95ce787758b1a7037b7049b07d610214"
server_key_iv: "1cfe9ea1e53c6d2dfc8ca75a6d593319"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.470851 6794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:11.474951 6794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:11.475435 6794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:11.475610 6794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:11.476094 6794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:11.476902 6794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:11.476969 6794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:11.477030 6794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:11.477077 6794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:11.485350 6794 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:39369
I20260928 14:41:11.485373 6921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:39369 every 8 connection(s)
I20260928 14:41:11.486243 6794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:41:11.491516 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.487746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:32775 (local address 127.1.201.62:43833)
0928 14:41:11.487902 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.487906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.488812 (+ 906us) server_negotiation.cc:407] Connection header received
0928 14:41:11.490077 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.490080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.490127 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.490193 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.491181 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.491184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.491186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.491196 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.491214 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.491221 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.491288 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.491392 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.491424 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:41:11.492295 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.488075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:43833 (local address 127.1.201.3:32775)
0928 14:41:11.488687 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.488722 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:11.489884 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:11.490333 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:11.490339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:11.490442 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:11.490974 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:11.490975 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:11.491005 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:11.491058 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:11.491291 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:11.491296 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:11.491299 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:41:11.491406 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:41:11.491552 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260928 14:41:11.493242 6922 heartbeater.cc:347] Connected to a master server at 127.1.201.62:43833
I20260928 14:41:11.493439 6922 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:11.493825 6922 heartbeater.cc:511] Master 127.1.201.62:43833 requested a full tablet report, sending...
I20260928 14:41:11.494778 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 6794
I20260928 14:41:11.494823 6472 ts_manager.cc:194] Registered new tserver with Master: 6f87318b281943bead2c0371c0619f9c (127.1.201.3:39369)
I20260928 14:41:11.494889 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:11.495149 1828 external_mini_cluster.cc:1428] Setting key bfe4525d729b8d29515a639a574b283e
I20260928 14:41:11.496970 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:41:11.500193 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.498783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.1.201.62:43833)
0928 14:41:11.498910 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.498914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.499094 (+ 180us) server_negotiation.cc:407] Connection header received
0928 14:41:11.499183 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.499185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.499214 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.499268 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.499893 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.499895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.499896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.499904 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.499916 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.499921 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.499978 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.500018 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.500044 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:41:11.502650 6472 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:41:11.505226 6472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:41:11.515334 6930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.512769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:38919 (local address 127.0.0.1:40846)
0928 14:41:11.513362 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.513371 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:11.513442 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:11.514002 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:11.514010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:11.514078 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:11.514535 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:11.514537 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:11.514556 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:11.514614 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:11.515019 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:11.515028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:11.515033 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:41:11.515134 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:41:11.515193 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":70,"threads_started":1}
I20260928 14:41:11.515399 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.513108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.1.201.1:38919)
0928 14:41:11.513557 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.513568 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.513586 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:41:11.513634 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.513639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.513852 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.513989 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.514768 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.514774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.514786 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.514797 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.514840 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.514884 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.514979 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.515168 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.515243 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":111,"threads_started":1}
I20260928 14:41:11.515753 6933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.512418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:39369 (local address 127.0.0.1:49880)
0928 14:41:11.512945 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.512957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:11.513200 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:11.514668 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:11.514673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:11.514691 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:11.515117 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:11.515120 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:11.515131 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:11.515156 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:11.515516 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:11.515520 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:11.515523 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:41:11.515604 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:41:11.515639 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"spinlock_wait_cycles":352896,"thread_start_us":60,"threads_started":1}
I20260928 14:41:11.516314 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.512308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:46267 (local address 127.0.0.1:38540)
0928 14:41:11.512755 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.512794 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:11.513200 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:11.514005 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:11.514008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:11.514091 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:11.514755 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:11.514757 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:11.514767 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:11.514792 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:11.516088 (+ 1296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:11.516092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:11.516097 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:41:11.516150 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:41:11.516197 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"spinlock_wait_cycles":1792,"thread_start_us":76,"threads_started":1}
I20260928 14:41:11.516710 6932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.512411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.1.201.2:46267)
0928 14:41:11.513525 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.513555 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.513581 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:41:11.513634 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.513639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.513849 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.514942 (+ 1093us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.515779 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.515784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.515914 (+ 130us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.515925 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.515961 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.515985 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.516508 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.516545 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.516594 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":80,"threads_started":1}
I20260928 14:41:11.517539 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.512924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.1.201.3:39369)
0928 14:41:11.513662 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.513672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.513690 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:41:11.513752 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.513755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.513964 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.514082 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.515290 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.515295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.515307 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.515319 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.515353 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.515378 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.515471 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.516092 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.516155 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":111,"threads_started":1}
I20260928 14:41:11.518122 6594 tablet_service.cc:1467] Processing CreateTablet for tablet c8d505edda9b445d96c21446faac5422 (DEFAULT_TABLE table=test-table [id=33124ca6a4a549d393a7bb115a271607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:11.519153 6594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d505edda9b445d96c21446faac5422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:11.519096 6725 tablet_service.cc:1467] Processing CreateTablet for tablet c8d505edda9b445d96c21446faac5422 (DEFAULT_TABLE table=test-table [id=33124ca6a4a549d393a7bb115a271607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:11.519994 6725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d505edda9b445d96c21446faac5422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:11.520128 6856 tablet_service.cc:1467] Processing CreateTablet for tablet c8d505edda9b445d96c21446faac5422 (DEFAULT_TABLE table=test-table [id=33124ca6a4a549d393a7bb115a271607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:11.520973 6856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d505edda9b445d96c21446faac5422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:11.524828 6936 tablet_bootstrap.cc:492] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151: Bootstrap starting.
I20260928 14:41:11.526657 6936 tablet_bootstrap.cc:654] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:11.526926 6937 tablet_bootstrap.cc:492] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611: Bootstrap starting.
I20260928 14:41:11.527858 6938 tablet_bootstrap.cc:492] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: Bootstrap starting.
I20260928 14:41:11.528024 6936 log.cc:824] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:11.529016 6937 tablet_bootstrap.cc:654] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:11.529708 6938 tablet_bootstrap.cc:654] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:11.529724 6937 log.cc:824] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:11.529937 6936 tablet_bootstrap.cc:492] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151: No bootstrap required, opened a new log
I20260928 14:41:11.530119 6936 ts_tablet_manager.cc:1397] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:41:11.530483 6938 log.cc:824] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:11.531513 6937 tablet_bootstrap.cc:492] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611: No bootstrap required, opened a new log
I20260928 14:41:11.531675 6937 ts_tablet_manager.cc:1397] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:41:11.532128 6938 tablet_bootstrap.cc:492] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: No bootstrap required, opened a new log
I20260928 14:41:11.532302 6938 ts_tablet_manager.cc:1397] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:41:11.533550 6936 raft_consensus.cc:348] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd86cc2b17d491886c1965f7d2fa611" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46267 } } peers { permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 } } peers { permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 } }
I20260928 14:41:11.533792 6936 raft_consensus.cc:374] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:11.533862 6936 raft_consensus.cc:729] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240b6f7c9e5c4f3cb9cfa14850dea151, State: Initialized, Role: FOLLOWER
I20260928 14:41:11.534264 6936 consensus_queue.cc:260] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd86cc2b17d491886c1965f7d2fa611" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46267 } } peers { permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 } } peers { permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 } }
I20260928 14:41:11.534822 6660 heartbeater.cc:503] Master 127.1.201.62:43833 was elected leader, sending a full tablet report...
I20260928 14:41:11.535132 6936 ts_tablet_manager.cc:1428] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:41:11.535815 6937 raft_consensus.cc:348] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd86cc2b17d491886c1965f7d2fa611" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46267 } } peers { permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 } } peers { permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 } }
I20260928 14:41:11.536031 6937 raft_consensus.cc:374] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:11.536082 6937 raft_consensus.cc:729] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd86cc2b17d491886c1965f7d2fa611, State: Initialized, Role: FOLLOWER
I20260928 14:41:11.536331 6938 raft_consensus.cc:348] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd86cc2b17d491886c1965f7d2fa611" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46267 } } peers { permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 } } peers { permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 } }
I20260928 14:41:11.536547 6938 raft_consensus.cc:374] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:11.536602 6938 raft_consensus.cc:729] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f87318b281943bead2c0371c0619f9c, State: Initialized, Role: FOLLOWER
I20260928 14:41:11.536978 6938 consensus_queue.cc:260] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd86cc2b17d491886c1965f7d2fa611" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46267 } } peers { permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 } } peers { permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 } }
I20260928 14:41:11.537134 6937 consensus_queue.cc:260] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd86cc2b17d491886c1965f7d2fa611" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46267 } } peers { permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 } } peers { permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 } }
I20260928 14:41:11.537874 6938 ts_tablet_manager.cc:1428] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:41:11.537907 6937 ts_tablet_manager.cc:1428] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:41:11.537940 6791 heartbeater.cc:503] Master 127.1.201.62:43833 was elected leader, sending a full tablet report...
I20260928 14:41:11.538798 6922 heartbeater.cc:503] Master 127.1.201.62:43833 was elected leader, sending a full tablet report...
W20260928 14:41:11.581245 6792 tablet.cc:2369] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:41:11.670104 6944 raft_consensus.cc:482] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:41:11.670318 6944 raft_consensus.cc:504] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd86cc2b17d491886c1965f7d2fa611" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46267 } } peers { permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 } } peers { permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 } }
I20260928 14:41:11.671391 6944 leader_election.cc:288] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f87318b281943bead2c0371c0619f9c (127.1.201.3:39369), 240b6f7c9e5c4f3cb9cfa14850dea151 (127.1.201.1:38919)
I20260928 14:41:11.673138 6793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.671549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:39369 (local address 127.1.201.2:49067)
0928 14:41:11.671657 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.671675 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:11.671748 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:11.672095 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:11.672097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:11.672117 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:11.672545 (+ 428us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:11.672546 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:11.672555 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:11.672601 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:11.672871 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:11.672873 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:11.672876 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:41:11.672965 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:41:11.673006 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 14:41:11.673148 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.671723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:38919 (local address 127.1.201.2:49943)
0928 14:41:11.672006 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.672017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:11.672115 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:11.672311 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:11.672313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:11.672326 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:11.672665 (+ 339us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:11.672669 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:11.672680 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:11.672703 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:11.672942 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:11.672948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:11.672950 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:41:11.673017 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:41:11.673040 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":111,"threads_started":1}
I20260928 14:41:11.673167 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.671801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:49943 (local address 127.1.201.1:38919)
0928 14:41:11.671902 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.671905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.672130 (+ 225us) server_negotiation.cc:407] Connection header received
0928 14:41:11.672176 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.672178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.672209 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.672302 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.672813 (+ 511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.672819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.672824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.672838 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.672857 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.672866 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.672945 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.673036 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.673069 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:41:11.673161 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.671734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:49067 (local address 127.1.201.3:39369)
0928 14:41:11.671862 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.671867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.671881 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:41:11.671926 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.671929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.671969 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.672045 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.672721 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.672724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.672726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.672756 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.672778 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.672789 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.672881 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.672978 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.673042 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:41:11.673750 6614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d505edda9b445d96c21446faac5422" candidate_uuid: "3cd86cc2b17d491886c1965f7d2fa611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" is_pre_election: true
I20260928 14:41:11.673750 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d505edda9b445d96c21446faac5422" candidate_uuid: "3cd86cc2b17d491886c1965f7d2fa611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f87318b281943bead2c0371c0619f9c" is_pre_election: true
I20260928 14:41:11.674046 6876 raft_consensus.cc:2454] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd86cc2b17d491886c1965f7d2fa611 in term 0.
I20260928 14:41:11.674046 6614 raft_consensus.cc:2454] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd86cc2b17d491886c1965f7d2fa611 in term 0.
I20260928 14:41:11.674530 6681 leader_election.cc:302] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 240b6f7c9e5c4f3cb9cfa14850dea151, 3cd86cc2b17d491886c1965f7d2fa611, 6f87318b281943bead2c0371c0619f9c; no voters:
I20260928 14:41:11.674770 6944 raft_consensus.cc:2781] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:41:11.674846 6944 raft_consensus.cc:482] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:41:11.674887 6944 raft_consensus.cc:3037] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:11.675870 6944 raft_consensus.cc:504] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd86cc2b17d491886c1965f7d2fa611" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46267 } } peers { permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 } } peers { permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 } }
I20260928 14:41:11.676147 6944 leader_election.cc:288] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [CANDIDATE]: Term 1 election: Requested vote from peers 6f87318b281943bead2c0371c0619f9c (127.1.201.3:39369), 240b6f7c9e5c4f3cb9cfa14850dea151 (127.1.201.1:38919)
I20260928 14:41:11.676425 6614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d505edda9b445d96c21446faac5422" candidate_uuid: "3cd86cc2b17d491886c1965f7d2fa611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151"
I20260928 14:41:11.676425 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d505edda9b445d96c21446faac5422" candidate_uuid: "3cd86cc2b17d491886c1965f7d2fa611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f87318b281943bead2c0371c0619f9c"
I20260928 14:41:11.676530 6614 raft_consensus.cc:3037] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:11.676530 6876 raft_consensus.cc:3037] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:11.677680 6876 raft_consensus.cc:2454] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd86cc2b17d491886c1965f7d2fa611 in term 1.
I20260928 14:41:11.677680 6614 raft_consensus.cc:2454] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd86cc2b17d491886c1965f7d2fa611 in term 1.
I20260928 14:41:11.677977 6681 leader_election.cc:302] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 240b6f7c9e5c4f3cb9cfa14850dea151, 3cd86cc2b17d491886c1965f7d2fa611, 6f87318b281943bead2c0371c0619f9c; no voters:
I20260928 14:41:11.678164 6944 raft_consensus.cc:2781] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:11.678345 6944 raft_consensus.cc:686] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 1 LEADER]: Becoming Leader. State: Replica: 3cd86cc2b17d491886c1965f7d2fa611, State: Running, Role: LEADER
I20260928 14:41:11.678655 6944 consensus_queue.cc:237] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd86cc2b17d491886c1965f7d2fa611" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46267 } } peers { permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 } } peers { permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 } }
W20260928 14:41:11.679729 6661 tablet.cc:2369] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:41:11.681756 6471 catalog_manager.cc:5495] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cd86cc2b17d491886c1965f7d2fa611 (127.1.201.2). New cstate: current_term: 1 leader_uuid: "3cd86cc2b17d491886c1965f7d2fa611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd86cc2b17d491886c1965f7d2fa611" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 } health_report { overall_health: UNKNOWN } } }
I20260928 14:41:11.703457 6932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.701973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38556 (local address 127.1.201.2:46267)
0928 14:41:11.702094 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.702098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.702111 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:41:11.702275 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.702278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.702339 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.702452 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.703091 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.703094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.703096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.703106 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.703123 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.703130 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.703233 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.703299 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.703333 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:41:11.708503 6876 raft_consensus.cc:1261] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c [term 1 FOLLOWER]: Refusing update from remote peer 3cd86cc2b17d491886c1965f7d2fa611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:11.708511 6614 raft_consensus.cc:1261] T c8d505edda9b445d96c21446faac5422 P 240b6f7c9e5c4f3cb9cfa14850dea151 [term 1 FOLLOWER]: Refusing update from remote peer 3cd86cc2b17d491886c1965f7d2fa611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:11.709112 6946 consensus_queue.cc:1035] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f87318b281943bead2c0371c0619f9c" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:11.709246 6944 consensus_queue.cc:1035] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "240b6f7c9e5c4f3cb9cfa14850dea151" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:11.716584 6949 mvcc.cc:205] Tried to move back new op lower bound from 7334324108111925248 to 7334324107998269440. Current Snapshot: MvccSnapshot[applied={T|T < 7334324108111925248}]
I20260928 14:41:11.716739 6951 mvcc.cc:205] Tried to move back new op lower bound from 7334324108111925248 to 7334324107998269440. Current Snapshot: MvccSnapshot[applied={T|T < 7334324108111925248}]
I20260928 14:41:11.723245 6471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:11.723383 6471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:11.725544 6471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 134145db8cbb420790975ab1751a77c4: Sending DeleteTablet for 3 replicas of tablet c8d505edda9b445d96c21446faac5422
I20260928 14:41:11.726220 6725 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d505edda9b445d96c21446faac5422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:11 UTC) from {username='slave'} at 127.0.0.1:38540
I20260928 14:41:11.726271 6856 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d505edda9b445d96c21446faac5422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:11 UTC) from {username='slave'} at 127.0.0.1:49880
I20260928 14:41:11.726378 6594 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d505edda9b445d96c21446faac5422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:11 UTC) from {username='slave'} at 127.0.0.1:40846
I20260928 14:41:11.726954 6959 tablet_replica.cc:330] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: stopping tablet replica
I20260928 14:41:11.727006 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 6532
I20260928 14:41:11.729002 6959 raft_consensus.cc:2229] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:11.729286 6959 raft_consensus.cc:2258] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:11.729624 6960 tablet_replica.cc:330] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611: stopping tablet replica
I20260928 14:41:11.729899 6960 raft_consensus.cc:2229] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:41:11.730185 6960 raft_consensus.cc:2258] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:11.731256 6959 ts_tablet_manager.cc:1905] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:11.733748 6959 ts_tablet_manager.cc:1918] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:11.733937 6959 log.cc:1196] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/c8d505edda9b445d96c21446faac5422
I20260928 14:41:11.734294 6960 ts_tablet_manager.cc:1905] T c8d505edda9b445d96c21446faac5422 P 3cd86cc2b17d491886c1965f7d2fa611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:11.734318 6959 ts_tablet_manager.cc:1939] T c8d505edda9b445d96c21446faac5422 P 6f87318b281943bead2c0371c0619f9c: Deleting consensus metadata
W20260928 14:41:11.735395 6459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:38919 (error 108)
I20260928 14:41:11.735622 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 6663
I20260928 14:41:11.735912 6458 catalog_manager.cc:4841] TS 6f87318b281943bead2c0371c0619f9c (127.1.201.3:39369): tablet c8d505edda9b445d96c21446faac5422 (table test-table [id=33124ca6a4a549d393a7bb115a271607]) successfully deleted
I20260928 14:41:11.737596 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.737348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:38919 (local address 127.0.0.1:40850)
0928 14:41:11.737456 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.737532 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:38919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 14:41:11.737833 6459 catalog_manager.cc:4584] TS 240b6f7c9e5c4f3cb9cfa14850dea151 (127.1.201.1:38919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8d505edda9b445d96c21446faac5422: Network error: Client connection negotiation failed: client connection to 127.1.201.1:38919: connect: Connection refused (error 111)
I20260928 14:41:11.741389 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 6794
I20260928 14:41:11.743009 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.741534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:46267 (local address 127.0.0.1:38568)
0928 14:41:11.742871 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.742931 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.2:46267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1245}
I20260928 14:41:11.746582 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 6441
2026-09-28T14:41:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (949 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 14:41:11.767479 1828 mini_oidc.cc:152] Starting JWKS server
I20260928 14:41:11.768996 1828 webserver.cc:460] Webserver started at https://127.0.0.1:37889/ using document root <none> and password file <none>
I20260928 14:41:11.769166 1828 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:41:11.769426 1828 webserver.cc:460] Webserver started at http://127.0.0.1:34793/ using document root <none> and password file <none>
2026-09-28T14:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:11Z Disabled control of system clock
I20260928 14:41:11.778138 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:33727
--webserver_interface=127.1.201.62
--webserver_port=0
--builtin_ntp_servers=127.1.201.20:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:33727
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37889/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/testchainca.cert with env {}
W20260928 14:41:11.879750 6968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:11.879966 6968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:11.880000 6968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:11.883821 6968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:11.883864 6968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:11.883879 6968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:11.883921 6968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:11.883956 6968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:41:11.883975 6968 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:41:11.884004 6968 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37889/jwks.json
I20260928 14:41:11.888691 6968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:33727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:33727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37889/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:11.889575 6968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:11.890448 6968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:11.898209 6973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:11.898186 6974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:11.898196 6976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:11.898504 6968 server_base.cc:1034] running on GCE node
I20260928 14:41:11.899117 6968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:11.900098 6968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:11.901302 6968 hybrid_clock.cc:648] HybridClock initialized: now 1790606471901283 us; error 53 us; skew 500 ppm
I20260928 14:41:11.903069 6968 webserver.cc:460] Webserver started at http://127.1.201.62:33013/ using document root <none> and password file <none>
I20260928 14:41:11.903630 6968 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:11.903728 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:11.903944 6968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:11.905884 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "00b904ea5bb74dffbf730f0a108c5c1b"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "9e4b679d7b2589ca2ba9c829c6ea5485"
server_key_iv: "4584ee57d2cb4619a01a00a6cc73ec53"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.906450 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "00b904ea5bb74dffbf730f0a108c5c1b"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "9e4b679d7b2589ca2ba9c829c6ea5485"
server_key_iv: "4584ee57d2cb4619a01a00a6cc73ec53"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.909874 6968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:41:11.911971 6982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:11.913040 6968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:11.913187 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "00b904ea5bb74dffbf730f0a108c5c1b"
format_stamp: "Formatted at 2026-09-28 14:41:11 on dist-test-slave-0n27"
server_key: "9e4b679d7b2589ca2ba9c829c6ea5485"
server_key_iv: "4584ee57d2cb4619a01a00a6cc73ec53"
server_key_version: "encryptionkey@0"
I20260928 14:41:11.913290 6968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:11.928785 6968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:11.929337 6968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:11.929510 6968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:11.935763 6968 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:33727
I20260928 14:41:11.935770 7034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:33727 every 8 connection(s)
I20260928 14:41:11.936601 6968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:11.939812 7035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:11.943100 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 6968
I20260928 14:41:11.943235 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:11.943481 1828 external_mini_cluster.cc:1428] Setting key b4614db7510fa3e00183e203ecc07eaf
I20260928 14:41:11.944962 7035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b: Bootstrap starting.
I20260928 14:41:11.946979 7035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:11.947659 7035 log.cc:824] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:11.949298 7035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b: No bootstrap required, opened a new log
I20260928 14:41:11.951915 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.944697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.1.201.62:33727)
0928 14:41:11.945298 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.945311 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.945342 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:41:11.945952 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.945969 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.946328 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.946628 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:11.947034 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:11.948037 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:11.948907 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:11.949379 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:11.949564 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:11.950410 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:11.950415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:11.950426 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:11.950777 (+ 351us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:11.950824 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:11.950842 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:11.950950 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:11.951104 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 14:41:11.951331 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260928 14:41:11.952795 7035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b904ea5bb74dffbf730f0a108c5c1b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33727 } }
I20260928 14:41:11.952975 7035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:11.953038 7035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b904ea5bb74dffbf730f0a108c5c1b, State: Initialized, Role: FOLLOWER
I20260928 14:41:11.953423 7035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b904ea5bb74dffbf730f0a108c5c1b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33727 } }
I20260928 14:41:11.953574 7035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:11.953644 7035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:11.953748 7035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:11.954568 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:41:11.954839 7035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b904ea5bb74dffbf730f0a108c5c1b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33727 } }
I20260928 14:41:11.955178 7035 leader_election.cc:302] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00b904ea5bb74dffbf730f0a108c5c1b; no voters:
I20260928 14:41:11.955473 7035 leader_election.cc:288] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:11.955727 7040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:11.955945 7040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [term 1 LEADER]: Becoming Leader. State: Replica: 00b904ea5bb74dffbf730f0a108c5c1b, State: Running, Role: LEADER
I20260928 14:41:11.956209 7040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b904ea5bb74dffbf730f0a108c5c1b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33727 } }
I20260928 14:41:11.956629 7035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:11.957294 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00b904ea5bb74dffbf730f0a108c5c1b. Latest consensus state: current_term: 1 leader_uuid: "00b904ea5bb74dffbf730f0a108c5c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b904ea5bb74dffbf730f0a108c5c1b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33727 } } }
I20260928 14:41:11.957417 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:11.957657 7041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00b904ea5bb74dffbf730f0a108c5c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b904ea5bb74dffbf730f0a108c5c1b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33727 } } }
I20260928 14:41:11.957756 7041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:11.957844 7048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:11.960666 7048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:11.965140 7048 catalog_manager.cc:1347] Generated new cluster ID: 632a649424064893b79caf21891580ac
I20260928 14:41:11.965196 7048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:11.974206 7048 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:11.974912 7048 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:11.977810 7048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00b904ea5bb74dffbf730f0a108c5c1b: Generated new TSK 0
I20260928 14:41:11.978358 7048 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:41:12.011044 7065 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 14:41:12.011142 7065 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:41:12.011631 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:11.994785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.1.201.62:33727)
0928 14:41:11.994915 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:11.994920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:11.995084 (+ 164us) server_negotiation.cc:407] Connection header received
0928 14:41:11.995270 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:11.995273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:11.995322 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:11.995419 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:41:11.995636 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:11.996428 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:11.997225 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:11.997587 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:11.997690 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:11.998027 (+ 337us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:41:12.011292 (+ 13265us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:41:12.011507 (+ 215us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53516: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 14:41:12.011700 7038 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53516: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 14:41:12.012596 7064 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:11.994650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33727 (local address 127.0.0.1:53516)
0928 14:41:11.995008 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:11.995020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:11.995115 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:11.995436 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:11.995439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:11.995445 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:41:11.995536 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:11.995539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:11.996531 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:11.996533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:11.997109 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:11.997115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:11.997695 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:11.997698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:11.997761 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:11.997849 (+ 88us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:41:12.011453 (+ 13604us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:41:12.011604 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.201.62:33727: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.201.62:33727: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 14:41:12.013383 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 6968
2026-09-28T14:41:12Z chronyd exiting
I20260928 14:41:12.050668 1828 test_util.cc:175] -----------------------------------------------
I20260928 14:41:12.050726 1828 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606444849692-1828-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (284 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 14:41:12.051687 1828 mini_oidc.cc:152] Starting JWKS server
I20260928 14:41:12.052467 1828 webserver.cc:460] Webserver started at https://127.0.0.1:40215/ using document root <none> and password file <none>
I20260928 14:41:12.052584 1828 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:41:12.052877 1828 webserver.cc:460] Webserver started at http://127.0.0.1:36171/ using document root <none> and password file <none>
2026-09-28T14:41:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:12Z Disabled control of system clock
I20260928 14:41:12.061064 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:32839
--webserver_interface=127.1.201.62
--webserver_port=0
--builtin_ntp_servers=127.1.201.20:38403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:32839
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40215/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/testchainca.cert with env {}
W20260928 14:41:12.159624 7071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:12.159807 7071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:12.159839 7071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:12.163152 7071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:12.163190 7071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:12.163203 7071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:12.163218 7071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:12.163229 7071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:41:12.163250 7071 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:41:12.163261 7071 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40215/jwks.json
I20260928 14:41:12.168015 7071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:38403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:32839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:32839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40215/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:12.168766 7071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:12.169554 7071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:12.176865 7079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:12.176841 7076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:12.176909 7071 server_base.cc:1034] running on GCE node
W20260928 14:41:12.176853 7077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:12.177449 7071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:12.178440 7071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:12.179596 7071 hybrid_clock.cc:648] HybridClock initialized: now 1790606472179580 us; error 27 us; skew 500 ppm
I20260928 14:41:12.181151 7071 webserver.cc:460] Webserver started at http://127.1.201.62:33951/ using document root <none> and password file <none>
I20260928 14:41:12.181653 7071 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:12.181734 7071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:12.181988 7071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:12.183805 7071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "2c6d464392e943d7854b389b15954a45"
format_stamp: "Formatted at 2026-09-28 14:41:12 on dist-test-slave-0n27"
server_key: "010328f334e8e5c495a259e11414e671"
server_key_iv: "35c4b9d73a1e1c68c1d7c7ff74dc76e5"
server_key_version: "encryptionkey@0"
I20260928 14:41:12.184293 7071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "2c6d464392e943d7854b389b15954a45"
format_stamp: "Formatted at 2026-09-28 14:41:12 on dist-test-slave-0n27"
server_key: "010328f334e8e5c495a259e11414e671"
server_key_iv: "35c4b9d73a1e1c68c1d7c7ff74dc76e5"
server_key_version: "encryptionkey@0"
I20260928 14:41:12.187392 7071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 14:41:12.189453 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:12.190825 7071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:12.190945 7071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "2c6d464392e943d7854b389b15954a45"
format_stamp: "Formatted at 2026-09-28 14:41:12 on dist-test-slave-0n27"
server_key: "010328f334e8e5c495a259e11414e671"
server_key_iv: "35c4b9d73a1e1c68c1d7c7ff74dc76e5"
server_key_version: "encryptionkey@0"
I20260928 14:41:12.191041 7071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:12.211858 7071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:12.212378 7071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:12.212504 7071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:12.218946 7071 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:32839
I20260928 14:41:12.218955 7137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:32839 every 8 connection(s)
I20260928 14:41:12.219784 7071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:12.222541 7138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:12.226580 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 7071
I20260928 14:41:12.226665 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:12.226941 1828 external_mini_cluster.cc:1428] Setting key 2b2902d91ec2cfeebf8873cb3e3ecc5b
I20260928 14:41:12.227389 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45: Bootstrap starting.
I20260928 14:41:12.229755 7138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:12.230450 7138 log.cc:824] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:12.232111 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45: No bootstrap required, opened a new log
I20260928 14:41:12.235412 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:12.228184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.1.201.62:32839)
0928 14:41:12.228771 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:12.228796 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:12.228826 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:41:12.229403 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:12.229418 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:12.229824 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:12.230172 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:12.230594 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.231524 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.232413 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.232878 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.233042 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:12.233869 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:12.233875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:12.233896 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:12.234245 (+ 349us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:12.234295 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:12.234315 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:12.234420 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:12.234532 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 14:41:12.234785 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":115,"threads_started":1}
I20260928 14:41:12.235536 7138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d464392e943d7854b389b15954a45" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 32839 } }
I20260928 14:41:12.235801 7138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:12.235883 7138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6d464392e943d7854b389b15954a45, State: Initialized, Role: FOLLOWER
I20260928 14:41:12.236292 7138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d464392e943d7854b389b15954a45" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 32839 } }
I20260928 14:41:12.236433 7138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:12.236502 7138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:12.236620 7138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:12.237519 7138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d464392e943d7854b389b15954a45" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 32839 } }
I20260928 14:41:12.237910 7138 leader_election.cc:302] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c6d464392e943d7854b389b15954a45; no voters:
I20260928 14:41:12.238174 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:41:12.238216 7138 leader_election.cc:288] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:12.238263 7143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:12.238459 7143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [term 1 LEADER]: Becoming Leader. State: Replica: 2c6d464392e943d7854b389b15954a45, State: Running, Role: LEADER
I20260928 14:41:12.238734 7143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d464392e943d7854b389b15954a45" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 32839 } }
I20260928 14:41:12.239355 7138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:12.239782 7143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c6d464392e943d7854b389b15954a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d464392e943d7854b389b15954a45" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 32839 } } }
I20260928 14:41:12.239889 7143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:12.240367 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c6d464392e943d7854b389b15954a45. Latest consensus state: current_term: 1 leader_uuid: "2c6d464392e943d7854b389b15954a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d464392e943d7854b389b15954a45" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 32839 } } }
I20260928 14:41:12.240465 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45 [sys.catalog]: This master's current role is: LEADER
W20260928 14:41:12.244001 7158 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:41:12.244061 7158 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:41:12.244135 7147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:12.244741 7147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:12.249126 7147 catalog_manager.cc:1347] Generated new cluster ID: 6cd5f183b58c473aa2b518dd84267a8e
I20260928 14:41:12.249181 7147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:12.254616 7147 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:12.255714 7147 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:12.266413 7147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c6d464392e943d7854b389b15954a45: Generated new TSK 0
I20260928 14:41:12.266929 7147 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:41:12.296973 7168 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 14:41:12.297044 7168 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:41:12.297621 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:12.280280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.1.201.62:32839)
0928 14:41:12.280407 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:12.280412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:12.280595 (+ 183us) server_negotiation.cc:407] Connection header received
0928 14:41:12.280734 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:12.280736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:12.280775 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:12.280848 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:41:12.281069 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.281528 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.282729 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.282934 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.283006 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:12.283336 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:41:12.297322 (+ 13986us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:41:12.297528 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50424: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 14:41:12.297701 7141 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50424: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 14:41:12.297833 7167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:12.280231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:32839 (local address 127.0.0.1:50424)
0928 14:41:12.280508 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:12.280522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:12.280622 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:12.280869 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:12.280872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:12.280883 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:41:12.280978 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:12.280983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.281619 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.281622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:12.282572 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:12.282582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.283035 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.283039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:12.283151 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:12.283176 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:41:12.297488 (+ 14312us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:41:12.297647 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.201.62:32839: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
I20260928 14:41:12.298362 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 7071
2026-09-28T14:41:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (284 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 14:41:12.336292 1828 mini_oidc.cc:152] Starting JWKS server
I20260928 14:41:12.337000 1828 webserver.cc:460] Webserver started at https://127.0.0.1:40419/ using document root <none> and password file <none>
I20260928 14:41:12.337069 1828 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:41:12.337299 1828 webserver.cc:460] Webserver started at http://127.0.0.1:36199/ using document root <none> and password file <none>
2026-09-28T14:41:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:12Z Disabled control of system clock
I20260928 14:41:12.345703 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40719
--webserver_interface=127.1.201.62
--webserver_port=0
--builtin_ntp_servers=127.1.201.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:40719
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40419/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:41:12.446203 7174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:12.446384 7174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:12.446416 7174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:12.450122 7174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:12.450161 7174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:12.450176 7174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:12.450191 7174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:12.450204 7174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:41:12.450217 7174 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:41:12.450229 7174 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40419/jwks.json
I20260928 14:41:12.454423 7174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:40719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40419/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:12.455128 7174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:12.456184 7174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:12.463496 7179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:12.463541 7174 server_base.cc:1034] running on GCE node
W20260928 14:41:12.463479 7180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:12.463495 7182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:12.464144 7174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:12.465062 7174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:12.466233 7174 hybrid_clock.cc:648] HybridClock initialized: now 1790606472466217 us; error 30 us; skew 500 ppm
I20260928 14:41:12.467720 7174 webserver.cc:460] Webserver started at http://127.1.201.62:38391/ using document root <none> and password file <none>
I20260928 14:41:12.468267 7174 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:12.468351 7174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:12.468562 7174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:12.470283 7174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "9d298bf9a958464f99732cc4d9bb0cfe"
format_stamp: "Formatted at 2026-09-28 14:41:12 on dist-test-slave-0n27"
server_key: "513c144ddf0e255998073c25c78d04c5"
server_key_iv: "74d788dd8214f1ccd0857f5703e12793"
server_key_version: "encryptionkey@0"
I20260928 14:41:12.470763 7174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "9d298bf9a958464f99732cc4d9bb0cfe"
format_stamp: "Formatted at 2026-09-28 14:41:12 on dist-test-slave-0n27"
server_key: "513c144ddf0e255998073c25c78d04c5"
server_key_iv: "74d788dd8214f1ccd0857f5703e12793"
server_key_version: "encryptionkey@0"
I20260928 14:41:12.473691 7174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:41:12.475670 7188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:12.476611 7174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:12.476747 7174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "9d298bf9a958464f99732cc4d9bb0cfe"
format_stamp: "Formatted at 2026-09-28 14:41:12 on dist-test-slave-0n27"
server_key: "513c144ddf0e255998073c25c78d04c5"
server_key_iv: "74d788dd8214f1ccd0857f5703e12793"
server_key_version: "encryptionkey@0"
I20260928 14:41:12.476843 7174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:12.500859 7174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:12.501355 7174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:12.501521 7174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:12.507167 7174 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:40719
I20260928 14:41:12.507179 7240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:40719 every 8 connection(s)
I20260928 14:41:12.507942 7174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:12.510370 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 7174
I20260928 14:41:12.510491 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:12.510705 7241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:12.510720 1828 external_mini_cluster.cc:1428] Setting key 7b163e67f5240f73b22d160feda72eef
I20260928 14:41:12.516718 7241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe: Bootstrap starting.
I20260928 14:41:12.519070 7241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:12.519580 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:12.511857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.1.201.62:40719)
0928 14:41:12.512421 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:12.512436 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:12.512482 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:41:12.513067 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:12.513086 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:12.513483 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:12.513849 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:12.514213 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.515472 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.516265 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.516861 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.517000 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:12.517729 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:12.517738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:12.517753 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:12.518202 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:12.518264 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:12.518288 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:12.518382 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:12.518492 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 14:41:12.518728 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":94,"threads_started":1}
I20260928 14:41:12.519838 7241 log.cc:824] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:12.521564 7241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe: No bootstrap required, opened a new log
I20260928 14:41:12.521963 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:41:12.524781 7241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298bf9a958464f99732cc4d9bb0cfe" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40719 } }
I20260928 14:41:12.524947 7241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:12.525022 7241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d298bf9a958464f99732cc4d9bb0cfe, State: Initialized, Role: FOLLOWER
I20260928 14:41:12.525406 7241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298bf9a958464f99732cc4d9bb0cfe" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40719 } }
I20260928 14:41:12.525530 7241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:12.525617 7241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:12.525717 7241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:12.526454 7241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298bf9a958464f99732cc4d9bb0cfe" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40719 } }
I20260928 14:41:12.526742 7241 leader_election.cc:302] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d298bf9a958464f99732cc4d9bb0cfe; no voters:
I20260928 14:41:12.526984 7241 leader_election.cc:288] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:12.527088 7246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:12.527319 7246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [term 1 LEADER]: Becoming Leader. State: Replica: 9d298bf9a958464f99732cc4d9bb0cfe, State: Running, Role: LEADER
I20260928 14:41:12.527644 7246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298bf9a958464f99732cc4d9bb0cfe" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40719 } }
I20260928 14:41:12.528208 7241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:12.529228 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d298bf9a958464f99732cc4d9bb0cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298bf9a958464f99732cc4d9bb0cfe" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40719 } } }
I20260928 14:41:12.529224 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d298bf9a958464f99732cc4d9bb0cfe. Latest consensus state: current_term: 1 leader_uuid: "9d298bf9a958464f99732cc4d9bb0cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298bf9a958464f99732cc4d9bb0cfe" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40719 } } }
I20260928 14:41:12.529357 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:12.529369 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:12.529724 7255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:12.532151 7255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:12.536713 7255 catalog_manager.cc:1347] Generated new cluster ID: 090f68c458804e68828cec678f120674
I20260928 14:41:12.536767 7255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:12.550825 7255 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:12.551766 7255 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:12.557061 7255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d298bf9a958464f99732cc4d9bb0cfe: Generated new TSK 0
I20260928 14:41:12.557703 7255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:41:12.572638 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:12.567800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.1.201.62:40719)
0928 14:41:12.567924 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:12.567930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:12.568066 (+ 136us) server_negotiation.cc:407] Connection header received
0928 14:41:12.568195 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:12.568198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:12.568249 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:12.568345 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:41:12.568573 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.569490 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.570396 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.570713 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.570801 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:12.571046 (+ 245us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:41:12.572309 (+ 1263us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:41:12.572512 (+ 203us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35382: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 14:41:12.572697 7244 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35382: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 14:41:12.574376 7270 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:12.567676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40719 (local address 127.0.0.1:35382)
0928 14:41:12.567986 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:12.567997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:12.568083 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:12.568375 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:12.568377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:12.568384 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:41:12.568474 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:12.568478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.569630 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.569632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:12.570296 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:12.570303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.570828 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.570831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:12.570889 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:12.570909 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:41:12.572457 (+ 1548us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:41:12.574227 (+ 1770us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.201.62:40719: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1607,"thread_start_us":113,"threads_started":1}
I20260928 14:41:12.574632 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 7174
E20260928 14:41:12.583015 7271 webserver.cc:533] Webserver: sslize failed:
E20260928 14:41:12.583070 7271 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T14:41:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (276 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T14:41:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:12Z Disabled control of system clock
I20260928 14:41:12.621537 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:37473
--webserver_interface=127.1.201.62
--webserver_port=0
--builtin_ntp_servers=127.1.201.20:35811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:37473
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/testchainca.cert with env {}
W20260928 14:41:12.720521 7275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:12.720763 7275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:12.720832 7275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:12.724571 7275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:12.724643 7275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:12.724673 7275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:12.724723 7275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:12.724753 7275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:41:12.724803 7275 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:41:12.724835 7275 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 14:41:12.729596 7275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:37473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:37473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:12.730448 7275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:12.731405 7275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:12.738935 7281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:12.738998 7275 server_base.cc:1034] running on GCE node
W20260928 14:41:12.738953 7280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:12.738966 7283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:12.739635 7275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:12.740559 7275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:12.741710 7275 hybrid_clock.cc:648] HybridClock initialized: now 1790606472741689 us; error 30 us; skew 500 ppm
I20260928 14:41:12.743405 7275 webserver.cc:460] Webserver started at http://127.1.201.62:33029/ using document root <none> and password file <none>
I20260928 14:41:12.743865 7275 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:12.743916 7275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:12.744131 7275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:12.745900 7275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "f4168db3a73d472fbcef2f18f210436b"
format_stamp: "Formatted at 2026-09-28 14:41:12 on dist-test-slave-0n27"
server_key: "dc61a8f651f63154b8a9fb4e247b2c04"
server_key_iv: "9b62aca0b8f8c6fdc9a18c5c4ba06389"
server_key_version: "encryptionkey@0"
I20260928 14:41:12.746404 7275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "f4168db3a73d472fbcef2f18f210436b"
format_stamp: "Formatted at 2026-09-28 14:41:12 on dist-test-slave-0n27"
server_key: "dc61a8f651f63154b8a9fb4e247b2c04"
server_key_iv: "9b62aca0b8f8c6fdc9a18c5c4ba06389"
server_key_version: "encryptionkey@0"
I20260928 14:41:12.749688 7275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:41:12.751667 7289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:12.752709 7275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:12.752827 7275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "f4168db3a73d472fbcef2f18f210436b"
format_stamp: "Formatted at 2026-09-28 14:41:12 on dist-test-slave-0n27"
server_key: "dc61a8f651f63154b8a9fb4e247b2c04"
server_key_iv: "9b62aca0b8f8c6fdc9a18c5c4ba06389"
server_key_version: "encryptionkey@0"
I20260928 14:41:12.752911 7275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:12.761220 7275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:12.761693 7275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:12.761854 7275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:12.767292 7275 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:37473
I20260928 14:41:12.767335 7341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:37473 every 8 connection(s)
I20260928 14:41:12.768090 7275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:12.770644 7342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:12.775229 7342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b: Bootstrap starting.
I20260928 14:41:12.775844 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 7275
I20260928 14:41:12.775935 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:12.776175 1828 external_mini_cluster.cc:1428] Setting key f64b82dc7bdc1b7e9283d1640e51062e
I20260928 14:41:12.777822 7342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:12.778648 7342 log.cc:824] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:12.780556 7342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b: No bootstrap required, opened a new log
I20260928 14:41:12.784646 7342 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4168db3a73d472fbcef2f18f210436b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37473 } }
I20260928 14:41:12.784890 7342 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:12.784946 7342 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4168db3a73d472fbcef2f18f210436b, State: Initialized, Role: FOLLOWER
I20260928 14:41:12.785362 7342 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4168db3a73d472fbcef2f18f210436b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37473 } }
I20260928 14:41:12.785506 7342 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:12.785566 7342 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:12.785652 7342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:12.785913 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:12.777338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.1.201.62:37473)
0928 14:41:12.778013 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:12.778036 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:12.778081 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:41:12.778845 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:12.778866 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:12.779268 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:12.779573 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:12.779924 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.781284 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.782201 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.782902 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.783053 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:12.784078 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:12.784084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:12.784098 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:12.784535 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:12.784596 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:12.784619 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:12.784722 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:12.784814 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 14:41:12.785054 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
I20260928 14:41:12.786638 7342 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4168db3a73d472fbcef2f18f210436b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37473 } }
I20260928 14:41:12.786988 7342 leader_election.cc:302] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4168db3a73d472fbcef2f18f210436b; no voters:
I20260928 14:41:12.787302 7342 leader_election.cc:288] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:12.787380 7347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:12.787588 7347 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [term 1 LEADER]: Becoming Leader. State: Replica: f4168db3a73d472fbcef2f18f210436b, State: Running, Role: LEADER
I20260928 14:41:12.787874 7347 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4168db3a73d472fbcef2f18f210436b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37473 } }
I20260928 14:41:12.788482 7342 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:12.788697 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:41:12.788975 7347 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4168db3a73d472fbcef2f18f210436b. Latest consensus state: current_term: 1 leader_uuid: "f4168db3a73d472fbcef2f18f210436b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4168db3a73d472fbcef2f18f210436b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37473 } } }
I20260928 14:41:12.789077 7347 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:12.789414 7351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:12.789511 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4168db3a73d472fbcef2f18f210436b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4168db3a73d472fbcef2f18f210436b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37473 } } }
I20260928 14:41:12.789659 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:12.792264 7351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:12.797438 7351 catalog_manager.cc:1347] Generated new cluster ID: c13d836e77f140c6a5b73fdccf05146e
I20260928 14:41:12.797506 7351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:12.806193 7351 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:12.806915 7351 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:12.812304 7351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4168db3a73d472fbcef2f18f210436b: Generated new TSK 0
I20260928 14:41:12.812798 7351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:41:12.872598 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:12.820062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.1.201.62:37473)
0928 14:41:12.820163 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:12.820167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:12.820351 (+ 184us) server_negotiation.cc:407] Connection header received
0928 14:41:12.820509 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:12.820511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:12.820547 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:12.820638 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:41:12.820849 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.821313 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.822242 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.822474 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.822579 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:12.822860 (+ 281us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:41:12.872198 (+ 49338us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:41:12.872462 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45138: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 14:41:12.872670 7345 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45138: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 14:41:12.872709 7371 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:12.819977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:37473 (local address 127.0.0.1:45138)
0928 14:41:12.820277 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:12.820287 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:12.820391 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:12.820641 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:12.820644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:12.820650 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:41:12.820750 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:12.820755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.821434 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.821437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:12.822102 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:12.822110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:12.822585 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:12.822589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:12.822685 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:12.822707 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:41:12.872382 (+ 49675us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:41:12.872540 (+ 158us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.201.62:37473: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
I20260928 14:41:12.873472 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 7275
2026-09-28T14:41:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (278 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (110 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (107 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:41:13 dist-test-slave-0n27 krb5kdc[7383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:41:13 dist-test-slave-0n27 krb5kdc[7383](info): set up 2 sockets
Sep 28 14:41:13 dist-test-slave-0n27 krb5kdc[7383](info): commencing operation
krb5kdc: starting...
W20260928 14:41:14.050894 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:41:14 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:41:14.159868 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40857
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:14.259650 7396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:14.259895 7396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:14.259958 7396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:14.263439 7396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:14.263504 7396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:14.263537 7396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:14.263586 7396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:14.263618 7396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:14.268160 7396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:40857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:14.269008 7396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:14.269855 7396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:14.277011 7402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:14.277020 7401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:14.277103 7396 server_base.cc:1034] running on GCE node
W20260928 14:41:14.277025 7404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:14.277649 7396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:14.278651 7396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:14.279848 7396 hybrid_clock.cc:648] HybridClock initialized: now 1790606474279820 us; error 50 us; skew 500 ppm
Sep 28 14:41:14 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:14.287338 7396 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:41:14.288108 7396 webserver.cc:460] Webserver started at http://127.1.201.62:43401/ using document root <none> and password file <none>
I20260928 14:41:14.288555 7396 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:14.288607 7396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:14.288745 7396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:14.290522 7396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "0afc54e056564d28bcaf67563f966ce0"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "0094b126a5e76dc1e423894fb3650bd2"
server_key_iv: "3d2d9cb9d35d416601f9d0b0cd79f891"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.290966 7396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "0afc54e056564d28bcaf67563f966ce0"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "0094b126a5e76dc1e423894fb3650bd2"
server_key_iv: "3d2d9cb9d35d416601f9d0b0cd79f891"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.294070 7396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:41:14.296195 7411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:14.297297 7396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:14.297432 7396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "0afc54e056564d28bcaf67563f966ce0"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "0094b126a5e76dc1e423894fb3650bd2"
server_key_iv: "3d2d9cb9d35d416601f9d0b0cd79f891"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.297529 7396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:14.314682 7396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:14.316982 7396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:14.317099 7396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:14.323309 7396 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:40857
I20260928 14:41:14.323333 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:40857 every 8 connection(s)
I20260928 14:41:14.324077 7396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:14.324779 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 7396
I20260928 14:41:14.324868 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:14.325124 1828 external_mini_cluster.cc:1428] Setting key 2abe9b0c8fcd47ebce09a365994f21f8
I20260928 14:41:14.326788 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:14.332741 7464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0: Bootstrap starting.
Sep 28 14:41:14 dist-test-slave-0n27 krb5kdc[7383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:14.335119 7464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:14.335983 7464 log.cc:824] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:14.337605 7464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0: No bootstrap required, opened a new log
I20260928 14:41:14.338656 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:14.326377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.1.201.62:40857)
0928 14:41:14.326748 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:14.326765 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:14.326804 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:41:14.327495 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:14.327514 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:14.327897 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:14.328230 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:14.328883 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.330304 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.331088 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.331747 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.331915 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:14.334268 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:14.334273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:14.334287 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:14.334315 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:14.336174 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:14.336556 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:14.336561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:14.336565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:14.336662 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:14.336918 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:14.336921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:14.336923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:14.337270 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:14.337463 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:14.337778 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 14:41:14.338050 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20260928 14:41:14.340708 7464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc54e056564d28bcaf67563f966ce0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40857 } }
I20260928 14:41:14.340947 7464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:14.341020 7464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afc54e056564d28bcaf67563f966ce0, State: Initialized, Role: FOLLOWER
I20260928 14:41:14.341416 7464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc54e056564d28bcaf67563f966ce0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40857 } }
I20260928 14:41:14.341538 7464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:14.341609 7464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:14.341696 7464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:14.342406 7464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc54e056564d28bcaf67563f966ce0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40857 } }
I20260928 14:41:14.342708 7464 leader_election.cc:302] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0afc54e056564d28bcaf67563f966ce0; no voters:
I20260928 14:41:14.342954 7464 leader_election.cc:288] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:14.343073 7469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:14.343305 7469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [term 1 LEADER]: Becoming Leader. State: Replica: 0afc54e056564d28bcaf67563f966ce0, State: Running, Role: LEADER
I20260928 14:41:14.343626 7469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc54e056564d28bcaf67563f966ce0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40857 } }
I20260928 14:41:14.344004 7464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:14.345109 7470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0afc54e056564d28bcaf67563f966ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc54e056564d28bcaf67563f966ce0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40857 } } }
I20260928 14:41:14.345114 7471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0afc54e056564d28bcaf67563f966ce0. Latest consensus state: current_term: 1 leader_uuid: "0afc54e056564d28bcaf67563f966ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc54e056564d28bcaf67563f966ce0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40857 } } }
I20260928 14:41:14.345235 7471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:14.345237 7470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:14.345696 7479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:14.348404 7479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:14.352690 7479 catalog_manager.cc:1347] Generated new cluster ID: eb374326588b42418d17f71a4494fa75
I20260928 14:41:14.352746 7479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:14.360513 7479 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:14.361372 7479 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:14.369004 7479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0afc54e056564d28bcaf67563f966ce0: Generated new TSK 0
I20260928 14:41:14.369524 7479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:41:14.396981 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40857
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:14.495369 7490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:14.495559 7490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:14.495633 7490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:14.499502 7490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:14.499544 7490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:14.499621 7490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:14.504281 7490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:14.505097 7490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:14.505916 7490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:14.513883 7495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:14.513943 7496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:14.514170 7490 server_base.cc:1034] running on GCE node
W20260928 14:41:14.514238 7498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:14.514521 7490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:14.515048 7490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:14.516222 7490 hybrid_clock.cc:648] HybridClock initialized: now 1790606474516203 us; error 37 us; skew 500 ppm
Sep 28 14:41:14 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:14.518748 7490 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:41:14.519491 7490 webserver.cc:460] Webserver started at http://127.1.201.1:39093/ using document root <none> and password file <none>
I20260928 14:41:14.519969 7490 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:14.520063 7490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:14.520262 7490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:14.522045 7490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "421ef536fc3d48ceb01d41f3610f815c"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "76c1c535b875a21fa3bd804006a8b77c"
server_key_iv: "74f6319243ffcd2b0edc12f15790d570"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.522540 7490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "421ef536fc3d48ceb01d41f3610f815c"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "76c1c535b875a21fa3bd804006a8b77c"
server_key_iv: "74f6319243ffcd2b0edc12f15790d570"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.525686 7490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:41:14.527657 7505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:14.528759 7490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:14.528883 7490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "421ef536fc3d48ceb01d41f3610f815c"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "76c1c535b875a21fa3bd804006a8b77c"
server_key_iv: "74f6319243ffcd2b0edc12f15790d570"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.529002 7490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:14.560982 7490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:14.563360 7490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:14.563534 7490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:14.564020 7490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:14.564711 7490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:14.564779 7490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:14.564841 7490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:14.564886 7490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:14.572923 7490 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:36477
I20260928 14:41:14.572959 7618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:36477 every 8 connection(s)
I20260928 14:41:14.573658 7490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:41:14 dist-test-slave-0n27 krb5kdc[7383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:14.582820 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 7490
I20260928 14:41:14.582922 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:14.583235 1828 external_mini_cluster.cc:1428] Setting key 5cebef1f925f88358997aa6a2c829d56
I20260928 14:41:14.586630 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:14.575148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:37457 (local address 127.1.201.62:40857)
0928 14:41:14.575264 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:14.575268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:14.575913 (+ 645us) server_negotiation.cc:407] Connection header received
0928 14:41:14.577182 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:14.577185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:14.577226 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:14.577293 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:14.578331 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.578848 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.579849 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.580060 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.580143 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:14.582195 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:14.582197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:14.582199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:14.582218 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:14.583623 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:14.584475 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:14.584478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:14.584482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:14.584571 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:14.585206 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:14.585209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:14.585211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:14.585555 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:14.585667 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:14.586297 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 14:41:14.586420 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:41:14.588047 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:14.575389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40857 (local address 127.1.201.1:37457)
0928 14:41:14.575783 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:14.575816 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:14.577013 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:14.577430 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:14.577436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:14.577771 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:14.578212 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:14.578224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.578959 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.578961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:14.579752 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:14.579759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.580176 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.580178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:14.580300 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:14.580802 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:14.580816 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:14.582073 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:14.583723 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:14.583730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:14.583744 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:14.584026 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:14.584715 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:14.584721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:14.584725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:14.585080 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:14.585722 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:14.585730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:14.586022 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 14:41:14.586761 (+ 739us) client_negotiation.cc:240] Negotiation successful
0928 14:41:14.587063 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":130,"threads_started":1}
I20260928 14:41:14.589771 7619 heartbeater.cc:347] Connected to a master server at 127.1.201.62:40857
I20260928 14:41:14.590056 7619 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:14.590536 7619 heartbeater.cc:511] Master 127.1.201.62:40857 requested a full tablet report, sending...
I20260928 14:41:14.592216 7428 ts_manager.cc:194] Registered new tserver with Master: 421ef536fc3d48ceb01d41f3610f815c (127.1.201.1:36477)
I20260928 14:41:14.593875 7428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:37457
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:41:14.616184 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40857
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:14.718050 7624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:14.718293 7624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:14.718362 7624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:14.722036 7624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:14.722107 7624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:14.722211 7624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:14.726716 7624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:14.727566 7624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:14.728456 7624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:41:14.737174 7624 server_base.cc:1034] running on GCE node
W20260928 14:41:14.737260 7629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:14.737187 7630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:14.737133 7632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:14.737703 7624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:14.738312 7624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:14.739498 7624 hybrid_clock.cc:648] HybridClock initialized: now 1790606474739489 us; error 25 us; skew 500 ppm
Sep 28 14:41:14 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:14.741981 7624 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:41:14.742695 7624 webserver.cc:460] Webserver started at http://127.1.201.2:34367/ using document root <none> and password file <none>
I20260928 14:41:14.743165 7624 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:14.743219 7624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:14.743477 7624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:14.745292 7624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "56db6d555fbb4f39a83f33b042be1db2"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "43e3ffe8bb7a0cd74783f767a2854082"
server_key_iv: "918e7b0dcdb9c38fdea21a3fac0023fa"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.745883 7624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "56db6d555fbb4f39a83f33b042be1db2"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "43e3ffe8bb7a0cd74783f767a2854082"
server_key_iv: "918e7b0dcdb9c38fdea21a3fac0023fa"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.749142 7624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:41:14.751246 7639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:14.752257 7624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:14.752386 7624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "56db6d555fbb4f39a83f33b042be1db2"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "43e3ffe8bb7a0cd74783f767a2854082"
server_key_iv: "918e7b0dcdb9c38fdea21a3fac0023fa"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.752480 7624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:14.769127 7624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:14.771806 7624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:14.771978 7624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:14.772461 7624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:14.773321 7624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:14.773396 7624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:14.773454 7624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:14.773483 7624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:14.781924 7624 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:40875
I20260928 14:41:14.781958 7752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:40875 every 8 connection(s)
I20260928 14:41:14.782656 7624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:41:14 dist-test-slave-0n27 krb5kdc[7383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:14.791584 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 7624
I20260928 14:41:14.791707 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:14.791966 1828 external_mini_cluster.cc:1428] Setting key 69c9d5c2915026fd6da9dd4d88af6aa8
I20260928 14:41:14.794531 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:14.784137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:39643 (local address 127.1.201.62:40857)
0928 14:41:14.784264 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:14.784267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:14.784962 (+ 695us) server_negotiation.cc:407] Connection header received
0928 14:41:14.786098 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:14.786102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:14.786155 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:14.786250 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:14.787277 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.787807 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.788851 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.789084 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.789160 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:14.791161 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:14.791164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:14.791166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:14.791187 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:14.792553 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:14.793050 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:14.793054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:14.793055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:14.793138 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:14.793467 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:14.793471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:14.793473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:14.793649 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:14.793772 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:14.794260 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 14:41:14.794371 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:41:14.795419 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:14.784478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40857 (local address 127.1.201.2:39643)
0928 14:41:14.784840 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:14.784874 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:14.785919 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:14.786414 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:14.786419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:14.786737 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:14.787157 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:14.787168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.787914 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.787916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:14.788742 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:14.788750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.789196 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.789199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:14.789322 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:14.789751 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:14.789765 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:14.791030 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:14.792663 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:14.792667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:14.792686 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:14.792951 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:14.793246 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:14.793248 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:14.793250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:14.793374 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:14.793772 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:14.793780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:14.794024 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 14:41:14.794203 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 14:41:14.794478 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":94,"threads_started":1}
I20260928 14:41:14.796484 7753 heartbeater.cc:347] Connected to a master server at 127.1.201.62:40857
I20260928 14:41:14.796695 7753 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:14.797170 7753 heartbeater.cc:511] Master 127.1.201.62:40857 requested a full tablet report, sending...
I20260928 14:41:14.798411 7428 ts_manager.cc:194] Registered new tserver with Master: 56db6d555fbb4f39a83f33b042be1db2 (127.1.201.2:40875)
I20260928 14:41:14.799077 7428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:39643
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:41:14.818864 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40857
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:14.919559 7758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:14.919787 7758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:14.919847 7758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:14.923529 7758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:14.923601 7758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:14.923681 7758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:14.928340 7758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:14.929283 7758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:14.930254 7758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:14.939026 7766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:14.939131 7758 server_base.cc:1034] running on GCE node
W20260928 14:41:14.939055 7764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:14.939042 7763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:14.939632 7758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:14.940181 7758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:14.941344 7758 hybrid_clock.cc:648] HybridClock initialized: now 1790606474941320 us; error 35 us; skew 500 ppm
Sep 28 14:41:14 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:14.943683 7758 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:41:14.944422 7758 webserver.cc:460] Webserver started at http://127.1.201.3:41851/ using document root <none> and password file <none>
I20260928 14:41:14.944918 7758 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:14.944995 7758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:14.945206 7758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:14.946988 7758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "f37ae5e20670423bb862b541b596c1ab"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "36c7f85bad68d2ed211b28369e128207"
server_key_iv: "df88c432fa18d83bbac72de87088e49b"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.947523 7758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "f37ae5e20670423bb862b541b596c1ab"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "36c7f85bad68d2ed211b28369e128207"
server_key_iv: "df88c432fa18d83bbac72de87088e49b"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.951911 7758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:41:14.954862 7773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:14.956218 7758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:41:14.956337 7758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "f37ae5e20670423bb862b541b596c1ab"
format_stamp: "Formatted at 2026-09-28 14:41:14 on dist-test-slave-0n27"
server_key: "36c7f85bad68d2ed211b28369e128207"
server_key_iv: "df88c432fa18d83bbac72de87088e49b"
server_key_version: "encryptionkey@0"
I20260928 14:41:14.956429 7758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:14.971899 7758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:14.974229 7758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:14.974391 7758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:14.974869 7758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:14.975555 7758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:14.975616 7758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:14.975697 7758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:14.975739 7758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:14.983091 7758 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:41571
I20260928 14:41:14.983110 7886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:41571 every 8 connection(s)
I20260928 14:41:14.983799 7758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:41:14 dist-test-slave-0n27 krb5kdc[7383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:14.993955 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 7758
I20260928 14:41:14.994071 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:14.994315 1828 external_mini_cluster.cc:1428] Setting key 1cedd2718742f8c70b31021cb438a82d
I20260928 14:41:14.994705 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:14.985280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:56215 (local address 127.1.201.62:40857)
0928 14:41:14.985395 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:14.985398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:14.986105 (+ 707us) server_negotiation.cc:407] Connection header received
0928 14:41:14.987254 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:14.987257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:14.987299 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:14.987365 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:14.988347 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.988842 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.989894 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.990081 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.990151 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:14.991980 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:14.991983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:14.991984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:14.992003 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:14.992982 (+ 979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:14.993358 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:14.993360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:14.993361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:14.993421 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:14.993654 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:14.993656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:14.993657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:14.993838 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:14.993957 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:14.994409 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 14:41:14.994511 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:41:14.995366 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:14.985552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40857 (local address 127.1.201.3:56215)
0928 14:41:14.985986 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:14.986018 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:14.987085 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:14.987494 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:14.987499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:14.987802 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:14.988234 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:14.988245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.988938 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.988940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:14.989732 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:14.989739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:14.990197 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:14.990200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:14.990318 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:14.990720 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:14.990733 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:14.991857 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:14.993066 (+ 1209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:14.993069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:14.993078 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:14.993287 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:14.993487 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:14.993488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:14.993490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:14.993591 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:14.993954 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:14.993959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:14.994214 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 14:41:14.994370 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 14:41:14.994641 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":133,"threads_started":1}
I20260928 14:41:14.996347 7887 heartbeater.cc:347] Connected to a master server at 127.1.201.62:40857
I20260928 14:41:14.996542 7887 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:14.996924 7887 heartbeater.cc:511] Master 127.1.201.62:40857 requested a full tablet report, sending...
I20260928 14:41:14.997874 7427 ts_manager.cc:194] Registered new tserver with Master: f37ae5e20670423bb862b541b596c1ab (127.1.201.3:41571)
I20260928 14:41:14.998384 7427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:56215
I20260928 14:41:15.007859 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/test-admin.keytab.
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:15.021997 1828 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 14:41:15.024490 7894 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:15.023133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.1.201.1:36477)
0928 14:41:15.023439 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:15.023448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:15.023468 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:41:15.023509 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:15.023513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:15.023666 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:15.023786 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:15.024330 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51676: BlockingRecv error: recv got EOF from 127.0.0.1:51676 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":125,"threads_started":1}
W20260928 14:41:15.024518 1828 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:15.025626 1828 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:15.026510 1828 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:41:15.034879 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:15.027286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.1.201.1:36477)
0928 14:41:15.027385 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:15.027389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:15.027477 (+ 88us) server_negotiation.cc:407] Connection header received
0928 14:41:15.027614 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:15.027623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:15.027658 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:15.027737 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:15.028237 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:15.028979 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:15.029757 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:15.030122 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:15.030236 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:15.032038 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:15.032046 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:15.032056 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:15.032075 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:15.033413 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:15.033750 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:15.033755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:15.033756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:15.033873 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:15.034080 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:15.034082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:15.034083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:15.034284 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:15.034427 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:15.034606 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 14:41:15.034757 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 14:41:15.037922 7894 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:15.037037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.1.201.1:36477)
0928 14:41:15.037141 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:15.037144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:15.037239 (+ 95us) server_negotiation.cc:407] Connection header received
0928 14:41:15.037360 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:15.037363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:15.037399 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:15.037467 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:15.037841 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51692: BlockingRecv error: recv got EOF from 127.0.0.1:51692 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 14:41:15.038048 1828 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:15.039036 1828 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:15.039796 1828 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:41:15.047194 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:15.040378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.1.201.1:36477)
0928 14:41:15.040476 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:15.040479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:15.040566 (+ 87us) server_negotiation.cc:407] Connection header received
0928 14:41:15.040672 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:15.040674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:15.040704 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:15.040781 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:15.041174 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:15.041601 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:15.042324 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:15.042551 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:15.042649 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:15.044611 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:15.044614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:15.044615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:15.044633 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:15.045964 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:15.046320 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:15.046322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:15.046323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:15.046393 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:15.046578 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:15.046580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:15.046581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:15.046722 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:15.046801 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:15.046965 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 14:41:15.047064 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 14:41:15.049456 7894 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:15.048577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.1.201.1:36477)
0928 14:41:15.048695 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:15.048699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:15.048767 (+ 68us) server_negotiation.cc:407] Connection header received
0928 14:41:15.048873 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:15.048876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:15.048915 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:15.048995 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:15.049359 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51696: BlockingRecv error: recv got EOF from 127.0.0.1:51696 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 14:41:15.049641 1828 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606444849692-1828-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:15.050518 1828 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:15.051287 1828 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:41:15.058015 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:15.051877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.1.201.1:36477)
0928 14:41:15.051994 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:15.051996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:15.052033 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:41:15.052103 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:15.052104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:15.052130 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:15.052183 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:15.052559 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:15.052966 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:15.053557 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:15.053727 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:15.053841 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:15.055499 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:15.055502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:15.055503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:15.055520 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:15.056753 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:15.057057 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:15.057060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:15.057061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:15.057119 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:15.057319 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:15.057320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:15.057321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:15.057433 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:15.057510 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:15.057715 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 14:41:15.057879 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:41:15.058923 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 7490
I20260928 14:41:15.064819 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 7624
I20260928 14:41:15.069552 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 7758
I20260928 14:41:15.074262 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 7396
2026-09-28T14:41:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1984 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7907](info): set up 2 sockets
Sep 28 14:41:15 dist-test-slave-0n27 krb5kdc[7907](info): commencing operation
krb5kdc: starting...
W20260928 14:41:16.034765 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:41:16 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.62.keytab.
Entry for principal oryx/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.62.keytab.
I20260928 14:41:16.142099 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:35941
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.62.keytab
--principal=oryx/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:35941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:16.241119 7920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:16.241376 7920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:16.241442 7920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:16.245065 7920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:16.245141 7920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:16.245177 7920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:16.245215 7920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:16.245245 7920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:16.250016 7920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:35941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.62.keytab
--principal=oryx/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:35941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:16.250837 7920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:16.251650 7920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:16.259029 7926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:16.259025 7925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:16.259027 7928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:16.259200 7920 server_base.cc:1034] running on GCE node
I20260928 14:41:16.259820 7920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:16.260731 7920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:16.261889 7920 hybrid_clock.cc:648] HybridClock initialized: now 1790606476261876 us; error 28 us; skew 500 ppm
Sep 28 14:41:16 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:16.264233 7920 init.cc:377] Logged in from keytab as oryx/127.1.201.62@KRBTEST.COM (short username oryx)
I20260928 14:41:16.264948 7920 webserver.cc:460] Webserver started at http://127.1.201.62:43555/ using document root <none> and password file <none>
I20260928 14:41:16.265388 7920 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:16.265465 7920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:16.265654 7920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:16.267426 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "5402e44e06f74daeb4a70d506780619a"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "7a477f4de25b441aa21c7e15d76e8077"
server_key_iv: "1b952b254a923a31f92b5f6d3618b0f0"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.267920 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "5402e44e06f74daeb4a70d506780619a"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "7a477f4de25b441aa21c7e15d76e8077"
server_key_iv: "1b952b254a923a31f92b5f6d3618b0f0"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.271121 7920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:16.273025 7935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:16.274084 7920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:16.274217 7920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "5402e44e06f74daeb4a70d506780619a"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "7a477f4de25b441aa21c7e15d76e8077"
server_key_iv: "1b952b254a923a31f92b5f6d3618b0f0"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.274312 7920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:16.300101 7920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:16.302356 7920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:16.302536 7920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:16.308739 7920 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:35941
I20260928 14:41:16.309002 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:35941 every 8 connection(s)
I20260928 14:41:16.309588 7920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:16.311826 7988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:16.316286 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: Bootstrap starting.
I20260928 14:41:16.316885 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 7920
I20260928 14:41:16.317034 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:16.317308 1828 external_mini_cluster.cc:1428] Setting key 506d5567c8716e308836543ffd44aa5d
I20260928 14:41:16.318899 7988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:16.319545 7988 log.cc:824] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:16.321372 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: No bootstrap required, opened a new log
Sep 28 14:41:16 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:16.325356 7988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } }
I20260928 14:41:16.325560 7988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:16.325618 7988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5402e44e06f74daeb4a70d506780619a, State: Initialized, Role: FOLLOWER
I20260928 14:41:16.326033 7988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } }
I20260928 14:41:16.326150 7988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:16.326234 7988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:16.326346 7988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:16.327121 7988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } }
I20260928 14:41:16.327440 7988 leader_election.cc:302] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5402e44e06f74daeb4a70d506780619a; no voters:
I20260928 14:41:16.327709 7988 leader_election.cc:288] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:16.327816 7993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:16.328063 7993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 1 LEADER]: Becoming Leader. State: Replica: 5402e44e06f74daeb4a70d506780619a, State: Running, Role: LEADER
I20260928 14:41:16.328421 7993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } }
I20260928 14:41:16.328930 7988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:16.330005 7994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } }
I20260928 14:41:16.330112 7994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:16.331269 7995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5402e44e06f74daeb4a70d506780619a. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } }
I20260928 14:41:16.331347 7995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:16.331473 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:16.318444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.1.201.62:35941)
0928 14:41:16.318816 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:16.318840 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:16.318871 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:41:16.319450 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:16.319464 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:16.319780 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:16.320101 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:16.320884 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.321860 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.322908 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.323307 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.323406 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:16.326105 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:16.326111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:16.326142 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:16.326173 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:16.328169 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:16.328658 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:16.328664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:16.328670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:16.328804 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:16.329108 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:16.329114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:16.329116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:16.329602 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:16.329812 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:16.330366 (+ 554us) server_negotiation.cc:299] Negotiation successful
0928 14:41:16.330653 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":128,"threads_started":1}
W20260928 14:41:16.333503 8008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:41:16.333556 8008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:41:16.333626 8002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:16.334391 8002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:16.339326 8002 catalog_manager.cc:1347] Generated new cluster ID: 01827db434a249b786859d1c0e3c185d
I20260928 14:41:16.339378 8002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:16.351998 8002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:16.353065 8002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:16.361418 8002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: Generated new TSK 0
I20260928 14:41:16.362005 8002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab.
Entry for principal oryx/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab.
I20260928 14:41:16.388855 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35941
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab
--principal=oryx/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:16.488039 8014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:16.488238 8014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:16.488277 8014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:16.491806 8014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:16.491843 8014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:16.491890 8014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:16.496469 8014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab
--principal=oryx/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:16.497171 8014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:16.498072 8014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:16.506582 8022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:16.506596 8020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:16.506594 8019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:16.506625 8014 server_base.cc:1034] running on GCE node
I20260928 14:41:16.507232 8014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:16.507781 8014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:16.508984 8014 hybrid_clock.cc:648] HybridClock initialized: now 1790606476508979 us; error 37 us; skew 500 ppm
Sep 28 14:41:16 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:16.511564 8014 init.cc:377] Logged in from keytab as oryx/127.1.201.1@KRBTEST.COM (short username oryx)
I20260928 14:41:16.512313 8014 webserver.cc:460] Webserver started at http://127.1.201.1:38957/ using document root <none> and password file <none>
I20260928 14:41:16.512781 8014 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:16.512833 8014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:16.513060 8014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:16.514813 8014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "bd640aa359054a81aa2e5011e31b16ec"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "d65f28e551ffaf650310a8a58b0214b8"
server_key_iv: "c0d9640c903195fa883aac9214b0f789"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.515306 8014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "bd640aa359054a81aa2e5011e31b16ec"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "d65f28e551ffaf650310a8a58b0214b8"
server_key_iv: "c0d9640c903195fa883aac9214b0f789"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.518396 8014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:41:16.520452 8029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:16.521418 8014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:41:16.521530 8014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "bd640aa359054a81aa2e5011e31b16ec"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "d65f28e551ffaf650310a8a58b0214b8"
server_key_iv: "c0d9640c903195fa883aac9214b0f789"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.521621 8014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:16.556456 8014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:16.559037 8014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:16.559242 8014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:16.559728 8014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:16.560434 8014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:16.560495 8014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:16.560554 8014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:16.560598 8014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:16.568394 8014 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:33547
I20260928 14:41:16.568425 8142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:33547 every 8 connection(s)
I20260928 14:41:16.569167 8014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:16.574007 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 8014
I20260928 14:41:16.574131 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:16.574398 1828 external_mini_cluster.cc:1428] Setting key fc7502cf7bd5854f293a828fa1283e92
Sep 28 14:41:16 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:16.581678 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:16.570653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:38583 (local address 127.1.201.62:35941)
0928 14:41:16.570776 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:16.570780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:16.571466 (+ 686us) server_negotiation.cc:407] Connection header received
0928 14:41:16.572639 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:16.572642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:16.572728 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:16.572807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:16.573861 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.574458 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.575685 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.575900 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.575998 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:16.578310 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:16.578313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:16.578316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:16.578346 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:16.579725 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:16.580273 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:16.580276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:16.580277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:16.580338 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:16.580712 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:16.580714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:16.580715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:16.580852 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:16.580916 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:16.581495 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 14:41:16.581575 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:41:16.582654 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:16.570960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.1:38583)
0928 14:41:16.571341 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:16.571377 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:16.572453 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:16.572975 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:16.572981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:16.573303 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:16.573710 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:16.573722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.574587 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.574590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:16.575536 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:16.575545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.576031 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.576034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:16.576205 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:16.576721 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:16.576758 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:16.578155 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:16.579829 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:16.579837 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:16.579851 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:16.580185 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:16.580420 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:16.580425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:16.580428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:16.580621 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:16.580948 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:16.580956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:16.581254 (+ 298us) client_negotiation.cc:769] Sending connection context
0928 14:41:16.581457 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 14:41:16.581751 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for oryx/127.1.201.2@KRBTEST.COM; defaulting to no policy
I20260928 14:41:16.583760 8143 heartbeater.cc:347] Connected to a master server at 127.1.201.62:35941
I20260928 14:41:16.583959 8143 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:16.584352 8143 heartbeater.cc:511] Master 127.1.201.62:35941 requested a full tablet report, sending...
I20260928 14:41:16.585654 7952 ts_manager.cc:194] Registered new tserver with Master: bd640aa359054a81aa2e5011e31b16ec (127.1.201.1:33547)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.2@KRBTEST.COM" created.
I20260928 14:41:16.586828 7952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:38583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab.
Entry for principal oryx/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab.
I20260928 14:41:16.599884 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35941
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab
--principal=oryx/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:16.696486 8148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:16.696674 8148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:16.696717 8148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:16.700416 8148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:16.700475 8148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:16.700527 8148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:16.705111 8148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab
--principal=oryx/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:16.705862 8148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:16.706612 8148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:16.714943 8156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:16.714950 8153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:16.714955 8154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:16.715283 8148 server_base.cc:1034] running on GCE node
I20260928 14:41:16.715708 8148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:16.716244 8148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:16.717430 8148 hybrid_clock.cc:648] HybridClock initialized: now 1790606476717421 us; error 27 us; skew 500 ppm
Sep 28 14:41:16 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:16.719976 8148 init.cc:377] Logged in from keytab as oryx/127.1.201.2@KRBTEST.COM (short username oryx)
I20260928 14:41:16.720705 8148 webserver.cc:460] Webserver started at http://127.1.201.2:43405/ using document root <none> and password file <none>
I20260928 14:41:16.721189 8148 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:16.721271 8148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:16.721473 8148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:16.723307 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "14f5ba50d8e54f0490dc670239d05163"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "b7aa71c9a12b2ad90827486f594e6a9e"
server_key_iv: "ef91deab8b542e4d7e8d9b5234fba7af"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.723799 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "14f5ba50d8e54f0490dc670239d05163"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "b7aa71c9a12b2ad90827486f594e6a9e"
server_key_iv: "ef91deab8b542e4d7e8d9b5234fba7af"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.727357 8148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:16.729822 8163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:16.731052 8148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:16.731184 8148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "14f5ba50d8e54f0490dc670239d05163"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "b7aa71c9a12b2ad90827486f594e6a9e"
server_key_iv: "ef91deab8b542e4d7e8d9b5234fba7af"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.731290 8148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:16.745702 8148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:16.748128 8148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:16.748313 8148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:16.748839 8148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:16.749567 8148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:16.749642 8148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:16.749701 8148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:16.749739 8148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:16.757331 8148 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:39669
I20260928 14:41:16.757352 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:39669 every 8 connection(s)
I20260928 14:41:16.758160 8148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:16.764920 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 8148
I20260928 14:41:16.765043 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:16.765327 1828 external_mini_cluster.cc:1428] Setting key 9d805be38b0100f3220d6245736440b4
Sep 28 14:41:16 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:16.770157 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:16.759706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:35823 (local address 127.1.201.62:35941)
0928 14:41:16.759844 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:16.759848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:16.760561 (+ 713us) server_negotiation.cc:407] Connection header received
0928 14:41:16.761734 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:16.761737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:16.761824 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:16.761901 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:16.762841 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.763346 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.764288 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.764481 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.764554 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:16.766804 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:16.766812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:16.766814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:16.766835 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:16.768218 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:16.768735 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:16.768739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:16.768741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:16.768821 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:16.769160 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:16.769164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:16.769166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:16.769346 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:16.769440 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:16.769896 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 14:41:16.769997 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:41:16.770756 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:16.760007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.2:35823)
0928 14:41:16.760427 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:16.760465 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:16.761578 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:16.762030 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:16.762036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:16.762347 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:16.762727 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:16.762739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.763440 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.763443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:16.764192 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:16.764200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.764596 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.764598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:16.764714 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:16.765203 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:16.765216 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:16.766648 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:16.768361 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:16.768366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:16.768378 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:16.768630 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:16.768915 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:16.768918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:16.768920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:16.769053 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:16.769439 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:16.769444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:16.769645 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 14:41:16.769844 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 14:41:16.770103 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":104,"threads_started":1}
I20260928 14:41:16.771795 8277 heartbeater.cc:347] Connected to a master server at 127.1.201.62:35941
I20260928 14:41:16.772011 8277 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:16.772487 8277 heartbeater.cc:511] Master 127.1.201.62:35941 requested a full tablet report, sending...
I20260928 14:41:16.773533 7952 ts_manager.cc:194] Registered new tserver with Master: 14f5ba50d8e54f0490dc670239d05163 (127.1.201.2:39669)
WARNING: no policy specified for oryx/127.1.201.3@KRBTEST.COM; defaulting to no policy
I20260928 14:41:16.774139 7952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:35823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab.
Entry for principal oryx/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab.
I20260928 14:41:16.790649 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35941
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab
--principal=oryx/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:16.891141 8282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:16.891374 8282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:16.891440 8282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:16.895057 8282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:16.895124 8282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:16.895203 8282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:16.899855 8282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab
--principal=oryx/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:16.900703 8282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:16.901506 8282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:16.909888 8287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:16.909914 8290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:16.909914 8288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:16.909968 8282 server_base.cc:1034] running on GCE node
I20260928 14:41:16.910498 8282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:16.911043 8282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:16.912218 8282 hybrid_clock.cc:648] HybridClock initialized: now 1790606476912189 us; error 41 us; skew 500 ppm
Sep 28 14:41:16 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:16.914667 8282 init.cc:377] Logged in from keytab as oryx/127.1.201.3@KRBTEST.COM (short username oryx)
I20260928 14:41:16.915436 8282 webserver.cc:460] Webserver started at http://127.1.201.3:37057/ using document root <none> and password file <none>
I20260928 14:41:16.915942 8282 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:16.916025 8282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:16.916224 8282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:16.918093 8282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "e1bc95a0d3cd4b339026c59818e3fadf"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "a76dd4b7892bbb52358dc6c9d00aeb6a"
server_key_iv: "70be97b509ac96a7b12777e6d30009a4"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.918613 8282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "e1bc95a0d3cd4b339026c59818e3fadf"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "a76dd4b7892bbb52358dc6c9d00aeb6a"
server_key_iv: "70be97b509ac96a7b12777e6d30009a4"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.921880 8282 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 14:41:16.923836 8297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:16.924952 8282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:16.925091 8282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "e1bc95a0d3cd4b339026c59818e3fadf"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "a76dd4b7892bbb52358dc6c9d00aeb6a"
server_key_iv: "70be97b509ac96a7b12777e6d30009a4"
server_key_version: "encryptionkey@0"
I20260928 14:41:16.925201 8282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:16.948544 8282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:16.950950 8282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:16.951124 8282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:16.951603 8282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:16.952293 8282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:16.952361 8282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:16.952427 8282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:16.952468 8282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:16.960294 8282 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:37881
I20260928 14:41:16.960330 8410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:37881 every 8 connection(s)
I20260928 14:41:16.960985 8282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:41:16.965497 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 8282
I20260928 14:41:16.965601 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:16.965920 1828 external_mini_cluster.cc:1428] Setting key 8d47fe9da30191781fa7ece3fa20c140
Sep 28 14:41:16 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:16.973016 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:16.962411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:39233 (local address 127.1.201.62:35941)
0928 14:41:16.962525 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:16.962529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:16.963163 (+ 634us) server_negotiation.cc:407] Connection header received
0928 14:41:16.964379 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:16.964382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:16.964439 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:16.964538 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:16.965728 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.966404 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.967478 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.967767 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.967853 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:16.969939 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:16.969942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:16.969944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:16.969966 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:16.971204 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:16.971575 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:16.971577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:16.971578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:16.971642 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:16.971919 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:16.971923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:16.971924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:16.972129 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:16.972221 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:16.972800 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 14:41:16.972896 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:41:16.973829 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:16.962691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.3:39233)
0928 14:41:16.963042 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:16.963076 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:16.964186 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:16.964690 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:16.964696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:16.965019 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:16.965554 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:16.965575 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.966508 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.966511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:16.967372 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:16.967380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:16.967891 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:16.967893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:16.968033 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:16.968462 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:16.968476 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:16.969765 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:16.971285 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:16.971288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:16.971297 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:16.971500 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:16.971710 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:16.971712 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:16.971713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:16.971823 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:16.972253 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:16.972260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:16.972538 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 14:41:16.972751 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 14:41:16.973068 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20260928 14:41:16.974731 8411 heartbeater.cc:347] Connected to a master server at 127.1.201.62:35941
I20260928 14:41:16.974951 8411 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:16.975348 8411 heartbeater.cc:511] Master 127.1.201.62:35941 requested a full tablet report, sending...
I20260928 14:41:16.976169 7952 ts_manager.cc:194] Registered new tserver with Master: e1bc95a0d3cd4b339026c59818e3fadf (127.1.201.3:37881)
I20260928 14:41:16.976644 7952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:39233
I20260928 14:41:16.979127 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.201.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.61.keytab.
Entry for principal oryx/127.1.201.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.61.keytab.
I20260928 14:41:17.004854 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.61:46839
--webserver_interface=127.1.201.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.61.keytab
--principal=oryx/127.1.201.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:35941,127.1.201.61:46839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:17.103030 8416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:17.103260 8416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:17.103328 8416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:17.107093 8416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:17.107161 8416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:17.107194 8416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:17.107251 8416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:17.107286 8416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:17.112092 8416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:35941,127.1.201.61:46839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.61.keytab
--principal=oryx/127.1.201.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.61:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.201.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
W20260928 14:41:17.112275 8416 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.201.62:35941,127.1.201.61:46839'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 14:41:17.112931 8416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:17.113735 8416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:17.121271 8424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:17.121330 8422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:17.121330 8421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:17.121434 8416 server_base.cc:1034] running on GCE node
I20260928 14:41:17.122022 8416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:17.122934 8416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:17.124104 8416 hybrid_clock.cc:648] HybridClock initialized: now 1790606477124089 us; error 24 us; skew 500 ppm
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:17.126516 8416 init.cc:377] Logged in from keytab as oryx/127.1.201.61@KRBTEST.COM (short username oryx)
I20260928 14:41:17.127247 8416 webserver.cc:460] Webserver started at http://127.1.201.61:44449/ using document root <none> and password file <none>
I20260928 14:41:17.127708 8416 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:17.127786 8416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:17.127978 8416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:17.129945 8416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data/instance:
uuid: "0511389fb9e14e169f05b776b7e40d5d"
format_stamp: "Formatted at 2026-09-28 14:41:17 on dist-test-slave-0n27"
server_key: "78e852d62a6dab5574cb6ff85d3e320b"
server_key_iv: "3ccf8a14751b76858ea59790c71cc71b"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.130477 8416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal/instance:
uuid: "0511389fb9e14e169f05b776b7e40d5d"
format_stamp: "Formatted at 2026-09-28 14:41:17 on dist-test-slave-0n27"
server_key: "78e852d62a6dab5574cb6ff85d3e320b"
server_key_iv: "3ccf8a14751b76858ea59790c71cc71b"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.133692 8416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:41:17.135814 8431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.136862 8416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:17.136982 8416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal
uuid: "0511389fb9e14e169f05b776b7e40d5d"
format_stamp: "Formatted at 2026-09-28 14:41:17 on dist-test-slave-0n27"
server_key: "78e852d62a6dab5574cb6ff85d3e320b"
server_key_iv: "3ccf8a14751b76858ea59790c71cc71b"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.137079 8416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:17.148447 8416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:17.150707 8416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:17.150911 8416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:17.156606 8416 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.61:46839
I20260928 14:41:17.156857 8483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.61:46839 every 8 connection(s)
I20260928 14:41:17.157424 8416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
I20260928 14:41:17.159485 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 8416
I20260928 14:41:17.159857 8484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:17.163388 8484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.201.61@KRBTEST.COM
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.61@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:17.172518 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.160686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.1.201.61:46839)
0928 14:41:17.161040 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.161052 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.161080 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:41:17.161607 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.161620 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.161959 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.162228 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.162919 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.163858 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.164671 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.165334 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.165483 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.167620 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.167625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.167635 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.167662 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.169582 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.170021 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.170031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.170035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.170127 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.170400 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.170403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.170405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.170809 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.171013 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.171353 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.171594 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":118,"threads_started":1}
I20260928 14:41:17.174659 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.164748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:60884)
0928 14:41:17.165099 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.165139 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.165250 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.165530 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.165535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.165935 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.166071 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.166077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.167273 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.167276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.168104 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.168115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.168637 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.168644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.168773 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.169187 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.169206 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.170546 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.172448 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.172454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.172458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.173318 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.173591 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.173593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.173594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.173715 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.174186 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.174195 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.174299 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.174422 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.174535 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":124,"threads_started":1}
I20260928 14:41:17.174681 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.164815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60884 (local address 127.1.201.62:35941)
0928 14:41:17.164964 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.164968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.165218 (+ 250us) server_negotiation.cc:407] Connection header received
0928 14:41:17.165345 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.165348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.165396 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.165487 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.166181 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.166634 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.168222 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.168521 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.168625 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.170685 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.170689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.170691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.170715 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.172340 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.173416 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.173420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.173421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.173506 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.173822 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.173825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.173826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.174073 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.174172 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.174473 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.174568 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:41:17.175733 8484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.61@KRBTEST.COM for oryx/127.1.201.61@KRBTEST.COM
I20260928 14:41:17.182757 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.176191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.1.201.61:46839)
0928 14:41:17.176270 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.176273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.176285 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:17.176347 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.176352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.176388 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.176469 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.176795 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.177553 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.178375 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.178588 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.178689 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.180212 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.180219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.180221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.180238 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.181216 (+ 978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.181593 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.181599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.181601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.181668 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.182013 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.182016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.182017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.182161 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.182262 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.182525 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.182635 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":23}
I20260928 14:41:17.182757 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.176103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46839 (local address 127.0.0.1:43638)
0928 14:41:17.176199 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.176213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.176280 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.176478 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.176480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.176621 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.176697 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.176702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.177651 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.177657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.178268 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.178276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.178693 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.178696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.178784 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.179128 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.179138 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.180090 (+ 952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.181310 (+ 1220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.181316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.181318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.181507 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.181765 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.181769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.181771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.181906 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.182263 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.182266 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.182345 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.182461 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.182584 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260928 14:41:17.185191 8484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: Bootstrap starting.
I20260928 14:41:17.187189 8484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:17.187829 8484 log.cc:824] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:17.189508 8484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: No bootstrap required, opened a new log
I20260928 14:41:17.192296 8484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } }
I20260928 14:41:17.192546 8484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:17.192620 8484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0511389fb9e14e169f05b776b7e40d5d, State: Initialized, Role: FOLLOWER
I20260928 14:41:17.193027 8484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } }
I20260928 14:41:17.193437 8490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } } }
I20260928 14:41:17.193534 8490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:41:17.194103 8484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:41:17.198141 8501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:41:17.198230 8501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:41:17.200155 8416 master_runner.cc:428] Detected that this master 0511389fb9e14e169f05b776b7e40d5d is joining an existing cluster
I20260928 14:41:17.200218 8416 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.201.61:46839
I20260928 14:41:17.209120 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.201959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.1.201.61:46839)
0928 14:41:17.202057 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.202062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.202456 (+ 394us) server_negotiation.cc:407] Connection header received
0928 14:41:17.202606 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.202610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.202644 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.202686 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.203229 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.203822 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.204654 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.204977 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.205059 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.206283 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.206289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.206291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.206316 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.207628 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.208010 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.208016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.208017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.208107 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.208363 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.208369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.208370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.208522 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.208616 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.208863 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.208998 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":35}
I20260928 14:41:17.209132 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.202136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46839 (local address 127.0.0.1:43648)
0928 14:41:17.202387 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.202401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.202498 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.202823 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.202825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.203014 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.203124 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.203130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.203915 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.203918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.204541 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.204550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.205107 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.205110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.205211 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.205674 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.205688 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.206159 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.207740 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.207744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.207746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.207924 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.208193 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.208198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.208200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.208277 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.208617 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.208620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.208712 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.208811 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.208946 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
I20260928 14:41:17.209681 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.201922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:60892)
0928 14:41:17.202168 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.202182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.202290 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.202567 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.202569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.202765 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.202860 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.202865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.203747 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.203750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.204626 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.204636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.205265 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.205267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.205335 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.205826 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.205840 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.206166 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.208067 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.208070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.208072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.208322 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.208667 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.208670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.208672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.208772 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.209213 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.209216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.209305 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.209398 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.209520 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260928 14:41:17.209815 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.201959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.1.201.62:35941)
0928 14:41:17.202235 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.202239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.202254 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:41:17.202429 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.202432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.202478 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.202558 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.202962 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.203636 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.204731 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.205060 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.205162 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.206357 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.206361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.206363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.206388 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.207923 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.208460 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.208464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.208466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.208553 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.208875 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.208878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.208880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.209075 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.209143 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.209483 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.209621 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.62@KRBTEST.COM for oryx/127.1.201.61@KRBTEST.COM
I20260928 14:41:17.221490 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.213464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46839 (local address 127.0.0.1:43658)
0928 14:41:17.213751 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.213814 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.213945 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.214229 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.214233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.214495 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.214575 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.214580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.215420 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.215422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.216002 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.216010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.216521 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.216523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.216589 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.217010 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.217019 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.218504 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.219947 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.219956 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.219961 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.220228 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.220449 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.220451 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.220453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.220537 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.220917 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.220925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.221033 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.221158 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.221337 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260928 14:41:17.221547 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.213483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.1.201.61:46839)
0928 14:41:17.213569 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.213572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.213898 (+ 326us) server_negotiation.cc:407] Connection header received
0928 14:41:17.214062 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.214065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.214122 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.214193 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.214669 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.215318 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.216102 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.216413 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.216516 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.218627 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.218630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.218631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.218651 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.219852 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.220312 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.220314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.220315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.220373 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.220612 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.220614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.220615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.220764 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.220861 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.221253 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.221390 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:41:17.222949 7952 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.201.61:46839: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d"
member_type: NON_VOTER
last_known_addr {
host: "127.1.201.61"
port: 46839
}
attrs {
promote: true
}
}
dest_uuid: "5402e44e06f74daeb4a70d506780619a"
cas_config_opid_index: -1
I20260928 14:41:17.223539 7952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } }
I20260928 14:41:17.225274 8510 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0511389fb9e14e169f05b776b7e40d5d (127.1.201.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } } }
I20260928 14:41:17.225812 8458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:17.227027 8458 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 FOLLOWER]: Refusing update from remote peer 5402e44e06f74daeb4a70d506780619a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 14:41:17.227149 8510 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.201.61:46839
I20260928 14:41:17.227537 8416 master.cc:537] Master@127.1.201.61:46839 shutting down...
I20260928 14:41:17.228850 8511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } } }
I20260928 14:41:17.228955 8511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:17.228374 8510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:17.231810 8416 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 14:41:17.231837 7937 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 14:41:17.232056 8416 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:17.232167 8416 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: stopping tablet replica
W20260928 14:41:17.234819 7937 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a -> Peer 0511389fb9e14e169f05b776b7e40d5d (127.1.201.61:46839): Couldn't send request to peer 0511389fb9e14e169f05b776b7e40d5d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.1.201.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.60@KRBTEST.COM" created.
I20260928 14:41:17.246572 8416 master.cc:552] Master@127.1.201.61:46839 shutdown complete.
I20260928 14:41:17.246613 8416 master_runner.cc:305] Clearing existing system tablet
I20260928 14:41:17.246831 8416 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:17.246910 8416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:17.248365 8416 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:41:17.249202 8518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.249400 8416 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:17.249521 8416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:17.249580 8416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal
uuid: "0511389fb9e14e169f05b776b7e40d5d"
format_stamp: "Formatted at 2026-09-28 14:41:17 on dist-test-slave-0n27"
server_key: "78e852d62a6dab5574cb6ff85d3e320b"
server_key_iv: "3ccf8a14751b76858ea59790c71cc71b"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.251336 8416 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:17.253734 8416 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:41:17.253877 8416 log.cc:1196] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 14:41:17.254125 8416 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: Deleting consensus metadata
I20260928 14:41:17.254294 8416 master_runner.cc:315] Copying system tablet from 127.1.201.62:35941
I20260928 14:41:17.255344 8416 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: tablet copy: Beginning tablet copy session from remote peer at address 127.1.201.62:35941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.60.keytab.
Entry for principal oryx/127.1.201.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.60.keytab.
I20260928 14:41:17.256479 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.60:44017
--webserver_interface=127.1.201.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.60.keytab
--principal=oryx/127.1.201.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:35941,127.1.201.61:46839,127.1.201.60:44017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:41:17.266351 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.256098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:60908)
0928 14:41:17.256375 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.256391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.256492 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.258186 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.258190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.258414 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.258880 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.258894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.259960 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.259964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.260863 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.260873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.261531 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.261535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.261628 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.262216 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.262231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.262611 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.264696 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.264699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.264702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.264958 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.265290 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.265292 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.265294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.265378 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.265843 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.265846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.265935 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.266037 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.266168 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
I20260928 14:41:17.266458 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.257760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.1.201.62:35941)
0928 14:41:17.257915 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.257920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.257934 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:41:17.257978 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.257981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.258043 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.258127 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.259020 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.259829 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.260991 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.261407 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.261532 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.262760 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.262765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.262769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.262802 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.264576 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.265064 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.265070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.265073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.265189 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.265475 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.265480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.265483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.265718 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.265849 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.266117 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.266248 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 14:41:17.266876 7972 tablet_copy_service.cc:140] P 5402e44e06f74daeb4a70d506780619a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0511389fb9e14e169f05b776b7e40d5d ({username='oryx', principal='oryx/127.1.201.61@KRBTEST.COM'} at 127.0.0.1:60908)
I20260928 14:41:17.267058 7972 tablet_copy_service.cc:161] P 5402e44e06f74daeb4a70d506780619a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0511389fb9e14e169f05b776b7e40d5d at {username='oryx', principal='oryx/127.1.201.61@KRBTEST.COM'} at 127.0.0.1:60908: session id = 0511389fb9e14e169f05b776b7e40d5d-00000000000000000000000000000000
I20260928 14:41:17.268781 7972 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:41:17.270431 8416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:41:17.272441 8416 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: tablet copy: Starting download of 0 data blocks...
I20260928 14:41:17.272609 8416 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: tablet copy: Starting download of 1 WAL segments...
I20260928 14:41:17.274108 8416 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:41:17.275717 7972 tablet_copy_service.cc:342] P 5402e44e06f74daeb4a70d506780619a: Request end of tablet copy session 0511389fb9e14e169f05b776b7e40d5d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.201.61@KRBTEST.COM'} at 127.0.0.1:60908
I20260928 14:41:17.275826 7972 tablet_copy_service.cc:433] P 5402e44e06f74daeb4a70d506780619a: ending tablet copy session 0511389fb9e14e169f05b776b7e40d5d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0511389fb9e14e169f05b776b7e40d5d
I20260928 14:41:17.283015 8416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:17.284696 8527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:17.284678 8526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:17.285081 8416 server_base.cc:1034] running on GCE node
W20260928 14:41:17.285910 8529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:17.286091 8416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:17.286329 8416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:17.287463 8416 hybrid_clock.cc:648] HybridClock initialized: now 1790606477287430 us; error 49 us; skew 500 ppm
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:17.288946 8416 init.cc:377] Logged in from keytab as oryx/127.1.201.61@KRBTEST.COM (short username oryx)
I20260928 14:41:17.289356 8416 webserver.cc:460] Webserver started at http://127.1.201.61:37727/ using document root <none> and password file <none>
I20260928 14:41:17.289526 8416 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:17.289556 8416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:17.290688 8416 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:17.291656 8536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.291888 8416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:17.291999 8416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal
uuid: "0511389fb9e14e169f05b776b7e40d5d"
format_stamp: "Formatted at 2026-09-28 14:41:17 on dist-test-slave-0n27"
server_key: "78e852d62a6dab5574cb6ff85d3e320b"
server_key_iv: "3ccf8a14751b76858ea59790c71cc71b"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.292116 8416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:17.338330 8416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:17.340544 8416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:17.345440 8416 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.61:46839
I20260928 14:41:17.345491 8588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.61:46839 every 8 connection(s)
I20260928 14:41:17.346168 8416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
I20260928 14:41:17.347940 8589 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:41:17.348938 8589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: Bootstrap starting.
I20260928 14:41:17.358415 8589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:41:17.358739 8589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: Bootstrap complete.
I20260928 14:41:17.359287 8589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } }
I20260928 14:41:17.359407 8589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0511389fb9e14e169f05b776b7e40d5d, State: Initialized, Role: LEARNER
I20260928 14:41:17.359498 8589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } }
I20260928 14:41:17.359774 8592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } } }
I20260928 14:41:17.359882 8589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:17.359916 8592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: This master's current role is: LEARNER
I20260928 14:41:17.362804 8603 catalog_manager.cc:1259] Loaded cluster ID: 01827db434a249b786859d1c0e3c185d
I20260928 14:41:17.362900 8603 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: loading cluster ID for follower catalog manager: success
I20260928 14:41:17.364893 8603 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: acquiring CA information for follower catalog manager: success
I20260928 14:41:17.366137 8603 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 14:41:17.379406 8525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:17.379638 8525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:17.379745 8525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:17.383709 8525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:17.383785 8525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:17.383812 8525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:17.383852 8525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:17.383886 8525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:17.388768 8525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:35941,127.1.201.61:46839,127.1.201.60:44017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.60.keytab
--principal=oryx/127.1.201.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.60:44017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.201.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:17.389564 8525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:17.390496 8525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:17.398126 8611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:17.398190 8525 server_base.cc:1034] running on GCE node
W20260928 14:41:17.398149 8608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:17.398140 8609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:17.398881 8525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:17.399778 8525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:17.400926 8525 hybrid_clock.cc:648] HybridClock initialized: now 1790606477400913 us; error 22 us; skew 500 ppm
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:17.403409 8525 init.cc:377] Logged in from keytab as oryx/127.1.201.60@KRBTEST.COM (short username oryx)
I20260928 14:41:17.404179 8525 webserver.cc:460] Webserver started at http://127.1.201.60:36897/ using document root <none> and password file <none>
I20260928 14:41:17.404676 8525 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:17.404729 8525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:17.404912 8525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:17.406756 8525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data/instance:
uuid: "999ad913e9c04ad581d6db34d5705d82"
format_stamp: "Formatted at 2026-09-28 14:41:17 on dist-test-slave-0n27"
server_key: "94946ad90acf0a7d38c76428c38b733c"
server_key_iv: "cff9ff915cc7f04442fe8bad90fb2820"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.407255 8525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal/instance:
uuid: "999ad913e9c04ad581d6db34d5705d82"
format_stamp: "Formatted at 2026-09-28 14:41:17 on dist-test-slave-0n27"
server_key: "94946ad90acf0a7d38c76428c38b733c"
server_key_iv: "cff9ff915cc7f04442fe8bad90fb2820"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.411588 8525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:41:17.413734 8618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.414860 8525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:17.414983 8525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal
uuid: "999ad913e9c04ad581d6db34d5705d82"
format_stamp: "Formatted at 2026-09-28 14:41:17 on dist-test-slave-0n27"
server_key: "94946ad90acf0a7d38c76428c38b733c"
server_key_iv: "cff9ff915cc7f04442fe8bad90fb2820"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.415100 8525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:17.426247 8525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:17.428912 8525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:17.429116 8525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:17.435984 8525 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.60:44017
I20260928 14:41:17.436237 8670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.60:44017 every 8 connection(s)
I20260928 14:41:17.436825 8525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
I20260928 14:41:17.439317 8671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:17.442601 8671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:41:17.444121 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 8525
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.60@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.201.60@KRBTEST.COM
I20260928 14:41:17.455967 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.444242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.1.201.62:35941)
0928 14:41:17.444349 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.444353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.445129 (+ 776us) server_negotiation.cc:407] Connection header received
0928 14:41:17.446097 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.446101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.446149 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.446242 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.447200 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.448009 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.449383 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.449715 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.449864 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.452271 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.452274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.452277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.452304 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.454031 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.454553 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.454557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.454560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.454653 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.454977 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.454982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.454984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.455153 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.455254 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.455703 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.455813 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:41:17.456661 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.444919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.1.201.60:44017)
0928 14:41:17.445343 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.445378 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.445407 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:41:17.445928 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.445947 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.446148 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.446353 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.447163 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.448313 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.449410 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.449997 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.450080 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.452746 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.452770 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.452784 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.452809 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.454539 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.454918 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.454923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.454925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.455004 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.455277 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.455281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.455283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.455503 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.455705 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.455935 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.456072 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"spinlock_wait_cycles":12032,"thread_start_us":146,"threads_started":1}
I20260928 14:41:17.456660 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.444546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56232)
0928 14:41:17.444992 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.445039 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.445918 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.446309 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.446315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.446728 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.447057 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.447075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.448145 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.448147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.449252 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.449261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.449915 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.449918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.450079 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.450610 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.450634 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.452129 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.454128 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.454135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.454141 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.454459 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.454756 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.454760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.454762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.454885 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.455251 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.455257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.455471 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.455646 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.455903 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"spinlock_wait_cycles":896,"thread_start_us":135,"threads_started":1}
I20260928 14:41:17.457871 8671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.60@KRBTEST.COM for oryx/127.1.201.61@KRBTEST.COM
I20260928 14:41:17.465232 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.458338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46839 (local address 127.0.0.1:57186)
0928 14:41:17.458417 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.458431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.458503 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.458883 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.458885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.458998 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.459071 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.459075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.460035 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.460037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.460569 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.460575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.461035 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.461037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.461115 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.461485 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.461495 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.462561 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.464128 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.464130 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.464131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.464293 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.464510 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.464511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.464513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.464574 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.464907 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.464908 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.464971 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.465055 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.465133 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 14:41:17.465359 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.458452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.1.201.61:46839)
0928 14:41:17.458694 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.458697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.458709 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:17.458744 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.458746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.458782 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.458854 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.459167 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.459941 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.460653 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.460934 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.461023 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.462687 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.462689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.462691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.462711 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.464045 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.464366 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.464368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.464369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.464433 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.464641 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.464643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.464644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.464816 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.464901 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.465106 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.465189 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260928 14:41:17.465926 8671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.60@KRBTEST.COM for oryx/127.1.201.60@KRBTEST.COM
I20260928 14:41:17.472123 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.466290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:44017 (local address 127.0.0.1:56814)
0928 14:41:17.466409 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.466418 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.466469 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.466627 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.466628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.466736 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.466789 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.466792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.467375 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.467377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.467820 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.467826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.468215 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.468217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.468269 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.468579 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.468589 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.469586 (+ 997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.471022 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.471025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.471026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.471204 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.471423 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.471425 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.471426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.471488 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.471828 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.471829 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.471889 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.471960 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.472037 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 14:41:17.472249 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.466338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.1.201.60:44017)
0928 14:41:17.466436 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.466440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.466452 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:17.466532 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.466534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.466563 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.466615 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.466863 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.467294 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.467901 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.468129 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.468196 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.469692 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.469695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.469697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.469715 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.470934 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.471290 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.471292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.471293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.471353 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.471567 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.471569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.471571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.471746 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.471829 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.472020 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.472108 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:41:17.474962 8671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Bootstrap starting.
I20260928 14:41:17.476929 8671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:17.477591 8671 log.cc:824] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:17.479290 8671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: No bootstrap required, opened a new log
I20260928 14:41:17.482015 8671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } }
I20260928 14:41:17.482208 8671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:17.482282 8671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999ad913e9c04ad581d6db34d5705d82, State: Initialized, Role: FOLLOWER
I20260928 14:41:17.482692 8671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } }
I20260928 14:41:17.483065 8677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } } }
I20260928 14:41:17.483160 8677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:41:17.483644 8671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:41:17.487690 8688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:41:17.487810 8688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:41:17.497439 8525 master_runner.cc:428] Detected that this master 999ad913e9c04ad581d6db34d5705d82 is joining an existing cluster
I20260928 14:41:17.497555 8525 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.201.60:44017
I20260928 14:41:17.508144 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.499994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.1.201.61:46839)
0928 14:41:17.500122 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.500127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.500649 (+ 522us) server_negotiation.cc:407] Connection header received
0928 14:41:17.500820 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.500823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.500862 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.500949 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.501423 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.502161 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.503389 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.503718 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.503791 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.505047 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.505051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.505053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.505075 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.506448 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.506891 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.506894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.506895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.506971 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.507263 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.507266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.507268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.507462 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.507532 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.507863 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.507974 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:41:17.508595 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.499930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46839 (local address 127.0.0.1:57202)
0928 14:41:17.500573 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.500586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.500708 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.500963 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.500966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.501141 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.501278 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.501284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.502276 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.502278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.503248 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.503258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.503947 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.503951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.504053 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.504618 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.504635 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.504920 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.506562 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.506565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.506567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.506796 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.507066 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.507068 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.507070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.507166 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.507595 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.507598 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.507692 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.508266 (+ 574us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.508416 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":62,"thread_start_us":113,"threads_started":1}
I20260928 14:41:17.509728 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 8014
I20260928 14:41:17.517407 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:33547
--local_ip_for_outbound_sockets=127.1.201.1
--tserver_master_addrs=127.1.201.62:35941
--webserver_port=38957
--webserver_interface=127.1.201.1
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab
--principal=oryx/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.201.62:35941,127.1.201.61:46839
--tserver_master_addrs=127.1.201.62:35941,127.1.201.61:46839,127.1.201.60:44017 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:41:17.517717 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.500632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.1.201.60:44017)
0928 14:41:17.500729 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.500735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.506646 (+ 5911us) server_negotiation.cc:407] Connection header received
0928 14:41:17.506971 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.506974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.507015 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.507059 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.507583 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.508281 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.509891 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.510267 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.510346 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.511524 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.511527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.511530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.511550 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.512847 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.513219 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.513224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.513226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.513296 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.513539 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.513542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.513544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.513759 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.513889 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.517375 (+ 3486us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.517491 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":48}
I20260928 14:41:17.518256 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.500362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:44017 (local address 127.0.0.1:56826)
0928 14:41:17.506579 (+ 6217us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.506592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.506703 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.507190 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.507195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.507383 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.507481 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.507490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.508608 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.508613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.509702 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.509712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.510474 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.510480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.510590 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.511002 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.511020 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.511398 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.512961 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.512964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.512966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.513132 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.513381 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.513383 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.513385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.513457 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.517069 (+ 3612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.517075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.517185 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.517850 (+ 665us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.517989 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5568,"thread_start_us":86,"threads_started":1}
I20260928 14:41:17.538130 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.499524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56234)
0928 14:41:17.499756 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.499770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.499857 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.500744 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.500748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.500962 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.501115 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.501124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.502183 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.502189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.503248 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.503258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.504275 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.504279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.504387 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.534253 (+ 29866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.534267 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.534676 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.536588 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.536591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.536592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.536813 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.537085 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.537088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.537090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.537190 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.537605 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.537608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.537699 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.537815 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.537942 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":89,"thread_start_us":86,"threads_started":1}
I20260928 14:41:17.538606 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.500251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.1.201.62:35941)
0928 14:41:17.500380 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.500386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.500403 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:41:17.500445 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.500450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.500502 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.500586 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.501250 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.502062 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.503389 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.503718 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.503802 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.534851 (+ 31049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.534855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.534858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.534883 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.536451 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.536916 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.536921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.536923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.536997 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.537283 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.537286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.537288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.537484 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.537564 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.538334 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.538443 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.62@KRBTEST.COM for oryx/127.1.201.60@KRBTEST.COM
I20260928 14:41:17.550112 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.541951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:44017 (local address 127.0.0.1:56828)
0928 14:41:17.542071 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.542084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.542150 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.542426 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.542429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.542584 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.542672 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.542678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.543604 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.543608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.544393 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.544404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.545001 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.545005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.545086 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.545519 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.545532 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.546890 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.548698 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.548701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.548703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.548929 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.549225 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.549227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.549229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.549322 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.549695 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.549697 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.549776 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.549879 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.549989 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:41:17.550544 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.542141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.1.201.60:44017)
0928 14:41:17.542233 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.542236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.542246 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:41:17.542298 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.542300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.542339 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.542405 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.542782 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.543478 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.544519 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.544816 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.544902 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.547022 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.547025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.547027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.547048 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.548576 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.549035 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.549039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.549040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.549114 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.549420 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.549422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.549424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.549580 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.549654 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.550342 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.550431 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:41:17.551263 7952 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.201.60:44017: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "999ad913e9c04ad581d6db34d5705d82"
member_type: NON_VOTER
last_known_addr {
host: "127.1.201.60"
port: 44017
}
attrs {
promote: true
}
}
dest_uuid: "5402e44e06f74daeb4a70d506780619a"
cas_config_opid_index: 5
I20260928 14:41:17.551782 7952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } }
I20260928 14:41:17.552871 8513 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 999ad913e9c04ad581d6db34d5705d82 (127.1.201.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.553285 8645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:17.554270 8513 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.201.60:44017
I20260928 14:41:17.554425 8513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.554513 8513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:17.555207 8645 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 FOLLOWER]: Refusing update from remote peer 5402e44e06f74daeb4a70d506780619a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:41:17.555538 8525 master.cc:537] Master@127.1.201.60:44017 shutting down...
I20260928 14:41:17.556047 8511 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Connected to new peer: Peer: permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:41:17.558665 7938 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a -> Peer 999ad913e9c04ad581d6db34d5705d82 (127.1.201.60:44017): Couldn't send request to peer 999ad913e9c04ad581d6db34d5705d82. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:41:17.560420 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.554776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46839 (local address 127.0.0.1:57204)
0928 14:41:17.556070 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.556082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.556150 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.556502 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.556505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.556520 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:17.556583 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.556588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.557494 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.557497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.558717 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.558730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.559986 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.559992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.560089 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.560135 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.560215 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.560284 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236}
I20260928 14:41:17.561026 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.554951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.1.201.61:46839)
0928 14:41:17.555947 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.555952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.556237 (+ 285us) server_negotiation.cc:407] Connection header received
0928 14:41:17.556281 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.556284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.556334 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.556405 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:17.556685 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.557375 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.558866 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.559622 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.559705 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.559856 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.560606 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.560845 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915}
I20260928 14:41:17.561472 8563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 LEARNER]: Refusing update from remote peer 5402e44e06f74daeb4a70d506780619a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:41:17.561919 8511 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 14:41:17.569430 8563 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 999ad913e9c04ad581d6db34d5705d82 (127.1.201.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.571593 8592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5402e44e06f74daeb4a70d506780619a. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.571710 8592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: This master's current role is: LEARNER
I20260928 14:41:17.571802 8592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: NON_VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: true } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.571864 8592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: This master's current role is: LEARNER
I20260928 14:41:17.572294 8525 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:17.572573 8525 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:17.572678 8525 tablet_replica.cc:330] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: stopping tablet replica
I20260928 14:41:17.573386 8511 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: attempting to promote NON_VOTER 0511389fb9e14e169f05b776b7e40d5d to VOTER
I20260928 14:41:17.573920 8511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } }
W20260928 14:41:17.575670 7938 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a -> Peer 999ad913e9c04ad581d6db34d5705d82 (127.1.201.60:44017): Couldn't send request to peer 999ad913e9c04ad581d6db34d5705d82. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:41:17.575899 8563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 LEARNER]: Refusing update from remote peer 5402e44e06f74daeb4a70d506780619a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 14:41:17.576263 8513 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 14:41:17.580106 8513 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0511389fb9e14e169f05b776b7e40d5d (127.1.201.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.581375 8513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.581475 8513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:17.581584 8513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.581655 8513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:17.582350 8563 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0511389fb9e14e169f05b776b7e40d5d (127.1.201.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.583660 8699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.583768 8699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:41:17.590597 8525 master.cc:552] Master@127.1.201.60:44017 shutdown complete.
I20260928 14:41:17.590646 8525 master_runner.cc:305] Clearing existing system tablet
I20260928 14:41:17.590849 8525 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:17.590899 8525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:17.592031 8525 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:41:17.592818 8701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.593024 8525 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:17.593152 8525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:41:17.593187 8525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal
uuid: "999ad913e9c04ad581d6db34d5705d82"
format_stamp: "Formatted at 2026-09-28 14:41:17 on dist-test-slave-0n27"
server_key: "94946ad90acf0a7d38c76428c38b733c"
server_key_iv: "cff9ff915cc7f04442fe8bad90fb2820"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.595003 8525 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:17.597133 8525 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:41:17.597214 8525 log.cc:1196] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 14:41:17.597435 8525 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Deleting consensus metadata
I20260928 14:41:17.597641 8525 master_runner.cc:315] Copying system tablet from 127.1.201.62:35941
I20260928 14:41:17.598786 8525 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: tablet copy: Beginning tablet copy session from remote peer at address 127.1.201.62:35941
I20260928 14:41:17.608243 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.599394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.1.201.62:35941)
0928 14:41:17.599531 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.599535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.600015 (+ 480us) server_negotiation.cc:407] Connection header received
0928 14:41:17.600188 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.600191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.600232 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.600275 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.600850 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.601584 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.602926 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.603264 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.603341 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.604789 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.604793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.604795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.604837 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.606455 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.606951 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.606955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.606957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.607038 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.607360 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.607365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.607367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.607572 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.607640 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.607953 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.608071 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:41:17.608774 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.599686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56250)
0928 14:41:17.599934 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.599947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.600080 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.600396 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.600399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.600589 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.600719 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.600726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.601713 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.601716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.602788 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.602800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.603489 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.603492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.603580 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.604120 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.604135 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.604541 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.606600 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.606603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.606605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.606835 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.607160 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.607164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.607167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.607258 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.607707 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.607709 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.607791 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.608405 (+ 614us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.608569 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":50,"threads_started":1}
I20260928 14:41:17.609143 7972 tablet_copy_service.cc:140] P 5402e44e06f74daeb4a70d506780619a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 999ad913e9c04ad581d6db34d5705d82 ({username='oryx', principal='oryx/127.1.201.60@KRBTEST.COM'} at 127.0.0.1:56250)
I20260928 14:41:17.609236 7972 tablet_copy_service.cc:161] P 5402e44e06f74daeb4a70d506780619a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 999ad913e9c04ad581d6db34d5705d82 at {username='oryx', principal='oryx/127.1.201.60@KRBTEST.COM'} at 127.0.0.1:56250: session id = 999ad913e9c04ad581d6db34d5705d82-00000000000000000000000000000000
I20260928 14:41:17.610319 7972 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:41:17.611398 8525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:41:17.613325 8525 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: tablet copy: Starting download of 0 data blocks...
I20260928 14:41:17.613488 8525 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: tablet copy: Starting download of 1 WAL segments...
I20260928 14:41:17.614658 8525 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:41:17.616189 7972 tablet_copy_service.cc:342] P 5402e44e06f74daeb4a70d506780619a: Request end of tablet copy session 999ad913e9c04ad581d6db34d5705d82-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.201.60@KRBTEST.COM'} at 127.0.0.1:56250
I20260928 14:41:17.616295 7972 tablet_copy_service.cc:433] P 5402e44e06f74daeb4a70d506780619a: ending tablet copy session 999ad913e9c04ad581d6db34d5705d82-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 999ad913e9c04ad581d6db34d5705d82
I20260928 14:41:17.624663 8525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:17.627023 8709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:17.627079 8711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:17.626992 8708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:17.627308 8525 server_base.cc:1034] running on GCE node
I20260928 14:41:17.627539 8525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:17.627810 8525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:17.631075 8525 hybrid_clock.cc:648] HybridClock initialized: now 1790606477631060 us; error 29 us; skew 500 ppm
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:17.632618 8525 init.cc:377] Logged in from keytab as oryx/127.1.201.60@KRBTEST.COM (short username oryx)
I20260928 14:41:17.633064 8525 webserver.cc:460] Webserver started at http://127.1.201.60:45171/ using document root <none> and password file <none>
I20260928 14:41:17.633255 8525 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:17.633296 8525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:17.634585 8525 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:17.637933 8718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.638163 8525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:41:17.638234 8525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal
uuid: "999ad913e9c04ad581d6db34d5705d82"
format_stamp: "Formatted at 2026-09-28 14:41:17 on dist-test-slave-0n27"
server_key: "94946ad90acf0a7d38c76428c38b733c"
server_key_iv: "cff9ff915cc7f04442fe8bad90fb2820"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.638314 8525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:17.658650 8525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:17.660758 8525 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260928 14:41:17.663821 8697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:17.664134 8697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:17.664245 8697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:41:17.664572 8525 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.60:44017
I20260928 14:41:17.665181 8525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
W20260928 14:41:17.669189 8697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:17.669291 8697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:17.669407 8697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:17.669514 8773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.60:44017 every 8 connection(s)
I20260928 14:41:17.671277 8774 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:41:17.672204 8774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Bootstrap starting.
I20260928 14:41:17.676343 8697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab
--principal=oryx/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:33547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=38957
--tserver_master_addrs=127.1.201.62:35941,127.1.201.61:46839,127.1.201.60:44017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:17.677388 8697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:17.678503 8697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:41:17.682965 8774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:41:17.683355 8774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Bootstrap complete.
I20260928 14:41:17.683878 8774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } }
I20260928 14:41:17.684073 8774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 999ad913e9c04ad581d6db34d5705d82, State: Initialized, Role: LEARNER
I20260928 14:41:17.684235 8774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } }
I20260928 14:41:17.684732 8774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:17.688570 8793 catalog_manager.cc:1259] Loaded cluster ID: 01827db434a249b786859d1c0e3c185d
I20260928 14:41:17.688714 8793 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: loading cluster ID for follower catalog manager: success
W20260928 14:41:17.690497 8779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:17.690507 8780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:17.691229 8793 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: acquiring CA information for follower catalog manager: success
I20260928 14:41:17.691676 8697 server_base.cc:1034] running on GCE node
W20260928 14:41:17.691851 8782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:17.692255 8697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:17.692883 8697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:17.691300 8778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:17.693035 8778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:41:17.693341 8793 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:41:17.694058 8697 hybrid_clock.cc:648] HybridClock initialized: now 1790606477694005 us; error 72 us; skew 500 ppm
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:17.696664 8697 init.cc:377] Logged in from keytab as oryx/127.1.201.1@KRBTEST.COM (short username oryx)
I20260928 14:41:17.697582 8697 webserver.cc:460] Webserver started at http://127.1.201.1:38957/ using document root <none> and password file <none>
I20260928 14:41:17.698169 8697 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:17.698269 8697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:17.701714 8697 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:17.703637 8800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.704949 8697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:17.705060 8697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "bd640aa359054a81aa2e5011e31b16ec"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "d65f28e551ffaf650310a8a58b0214b8"
server_key_iv: "c0d9640c903195fa883aac9214b0f789"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.705631 8697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:17.721611 8697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:17.724473 8697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:17.724645 8697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:17.725206 8697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:17.725992 8697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:17.726055 8697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.726125 8697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:17.726172 8697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.736989 8697 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:33547
I20260928 14:41:17.737016 8913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:33547 every 8 connection(s)
I20260928 14:41:17.737927 8697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:17.745730 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 8697
I20260928 14:41:17.745950 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 8148
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.61@KRBTEST.COM
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.60@KRBTEST.COM
I20260928 14:41:17.753662 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:39669
--local_ip_for_outbound_sockets=127.1.201.2
--tserver_master_addrs=127.1.201.62:35941
--webserver_port=43405
--webserver_interface=127.1.201.2
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab
--principal=oryx/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.201.62:35941,127.1.201.61:46839
--tserver_master_addrs=127.1.201.62:35941,127.1.201.61:46839,127.1.201.60:44017 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:41:17.755143 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.741139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:52613 (local address 127.1.201.61:46839)
0928 14:41:17.741468 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.741472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.741492 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:41:17.742429 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.742435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.742492 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.742543 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.744272 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.745010 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.747585 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.747917 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.747987 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.751557 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.751561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.751563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.751585 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.752983 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.753485 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.753488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.753490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.753558 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.754114 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.754117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.754119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.754311 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.754390 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.754883 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.754985 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260928 14:41:17.756158 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.740184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46839 (local address 127.1.201.1:52613)
0928 14:41:17.741199 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.741242 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.742151 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.742827 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.742834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.743407 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.744102 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.744119 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.745175 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.745177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.747428 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.747441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.748189 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.748192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.748390 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.748964 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.748985 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.751403 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.753094 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.753099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.753111 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.753392 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.753644 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.753647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.753649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.754018 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.754441 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.754446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.754658 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.755454 (+ 796us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.755572 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":261,"spinlock_wait_cycles":47232,"thread_start_us":111,"threads_started":1}
I20260928 14:41:17.756163 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.740184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:44017 (local address 127.1.201.1:49163)
0928 14:41:17.741940 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.741950 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.742160 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.742827 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.742835 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.743401 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.744104 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.744119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.745313 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.745316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.747662 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.747675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.748444 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.748451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.748578 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.749350 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.749368 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.751419 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.753745 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.753748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.753750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.754008 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.754296 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.754301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.754305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.754397 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.754838 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.754843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.754942 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.755052 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.755321 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1681,"mutex_wait_us":241,"spinlock_wait_cycles":32384,"thread_start_us":134,"threads_started":1}
I20260928 14:41:17.757118 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.741292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:49163 (local address 127.1.201.60:44017)
0928 14:41:17.741609 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.741611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.742029 (+ 418us) server_negotiation.cc:407] Connection header received
0928 14:41:17.742474 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.742477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.742532 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.742637 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.744257 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.745210 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.747791 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.748203 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.748314 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.751565 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.751571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.751576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.751615 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.753630 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.754115 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.754121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.754125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.754205 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.754480 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.754485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.754489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.754697 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.754792 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.756256 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.756377 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":61,"threads_started":1}
I20260928 14:41:17.757834 8916 heartbeater.cc:347] Connected to a master server at 127.1.201.61:46839
I20260928 14:41:17.758023 8916 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:17.758024 8915 heartbeater.cc:347] Connected to a master server at 127.1.201.60:44017
I20260928 14:41:17.758157 8915 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:17.758621 8916 heartbeater.cc:511] Master 127.1.201.61:46839 requested a full tablet report, sending...
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:17.758622 8915 heartbeater.cc:511] Master 127.1.201.60:44017 requested a full tablet report, sending...
I20260928 14:41:17.760093 8553 ts_manager.cc:194] Registered new tserver with Master: bd640aa359054a81aa2e5011e31b16ec (127.1.201.1:33547)
I20260928 14:41:17.761238 8738 ts_manager.cc:194] Registered new tserver with Master: bd640aa359054a81aa2e5011e31b16ec (127.1.201.1:33547)
I20260928 14:41:17.762853 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.741285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:33031 (local address 127.1.201.62:35941)
0928 14:41:17.741397 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.741403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.742321 (+ 918us) server_negotiation.cc:407] Connection header received
0928 14:41:17.742486 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.742489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.742538 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.742591 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.744257 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.744998 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.755837 (+ 10839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.756137 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.756215 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.759624 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.759627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.759630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.759654 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.761163 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.761662 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.761667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.761670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.761752 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.762087 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.762090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.762091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.762288 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.762380 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.762673 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.762758 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:41:17.762982 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.740796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.1:33031)
0928 14:41:17.742238 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.742250 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.742384 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.742827 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.742835 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.743401 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.744111 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.744119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.745166 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.745169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.755710 (+ 10541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.755720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.756368 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.756372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.756460 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.756972 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.756984 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.759470 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.761268 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.761271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.761273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.761549 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.761879 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.761883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.761886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.761970 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.762393 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.762399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.762502 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.762599 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.762740 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"mutex_wait_us":9725,"thread_start_us":80,"threads_started":1}
I20260928 14:41:17.763598 8914 heartbeater.cc:347] Connected to a master server at 127.1.201.62:35941
I20260928 14:41:17.763708 8914 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:17.763875 8914 heartbeater.cc:511] Master 127.1.201.62:35941 requested a full tablet report, sending...
I20260928 14:41:17.764596 7952 ts_manager.cc:194] Re-registered known tserver with Master: bd640aa359054a81aa2e5011e31b16ec (127.1.201.1:33547)
I20260928 14:41:17.765130 7952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:33031
W20260928 14:41:17.861713 8922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:17.861963 8922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:17.862025 8922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:17.865547 8922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:17.865587 8922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:17.865667 8922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:17.870496 8922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab
--principal=oryx/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:39669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=43405
--tserver_master_addrs=127.1.201.62:35941,127.1.201.61:46839,127.1.201.60:44017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:17.871313 8922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:17.872135 8922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:17.881258 8927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:17.881281 8928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:17.881290 8930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:17.881333 8922 server_base.cc:1034] running on GCE node
I20260928 14:41:17.881935 8922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:17.882489 8922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:17.883667 8922 hybrid_clock.cc:648] HybridClock initialized: now 1790606477883660 us; error 26 us; skew 500 ppm
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:17.886379 8922 init.cc:377] Logged in from keytab as oryx/127.1.201.2@KRBTEST.COM (short username oryx)
I20260928 14:41:17.887171 8922 webserver.cc:460] Webserver started at http://127.1.201.2:43405/ using document root <none> and password file <none>
I20260928 14:41:17.887655 8922 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:17.887723 8922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:17.891253 8922 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:17.893256 8937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.894388 8922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:17.894498 8922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "14f5ba50d8e54f0490dc670239d05163"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "b7aa71c9a12b2ad90827486f594e6a9e"
server_key_iv: "ef91deab8b542e4d7e8d9b5234fba7af"
server_key_version: "encryptionkey@0"
I20260928 14:41:17.895025 8922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:17.907186 8922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:17.909765 8922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:17.909996 8922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:17.910516 8922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:17.911216 8922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:17.911300 8922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.911363 8922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:17.911409 8922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:17.919829 8922 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:39669
I20260928 14:41:17.919849 9050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:39669 every 8 connection(s)
I20260928 14:41:17.920625 8922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:17.925891 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 8922
I20260928 14:41:17.926061 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 8282
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.60@KRBTEST.COM
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
Sep 28 14:41:17 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606477, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.61@KRBTEST.COM
I20260928 14:41:17.933841 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:37881
--local_ip_for_outbound_sockets=127.1.201.3
--tserver_master_addrs=127.1.201.62:35941
--webserver_port=37057
--webserver_interface=127.1.201.3
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab
--principal=oryx/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.201.62:35941,127.1.201.61:46839
--tserver_master_addrs=127.1.201.62:35941,127.1.201.61:46839,127.1.201.60:44017 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:41:17.936748 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.922192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:51605 (local address 127.1.201.60:44017)
0928 14:41:17.922333 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.922337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.923518 (+ 1181us) server_negotiation.cc:407] Connection header received
0928 14:41:17.924322 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.924326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.924374 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.924422 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.925641 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.926411 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.928288 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.928553 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.928670 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.932576 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.932582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.932588 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.932614 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.934369 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.934934 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.934941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.934945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.935030 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.935472 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.935476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.935477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.935705 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.935781 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.936429 (+ 648us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.936554 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:41:17.937490 9056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.922546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.2:53959)
0928 14:41:17.923400 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.923441 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.924146 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.924564 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.924566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.924991 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.925504 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.925513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.926574 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.926579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.928024 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.928035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.928842 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.928849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.928986 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.929672 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.929692 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.932393 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.935707 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.935712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.935714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.935928 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.936211 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.936215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.936217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.936305 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.936765 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.936769 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.936861 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.936976 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.937114 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":147,"spinlock_wait_cycles":161792,"thread_start_us":111,"threads_started":1}
I20260928 14:41:17.937700 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.922545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:44017 (local address 127.1.201.2:51605)
0928 14:41:17.923400 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.923441 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.924146 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.924564 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.924566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.924976 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.925504 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.925513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.927294 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.927297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.928172 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.928179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.928739 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.928748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.928949 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.929600 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.929619 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.932181 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.934495 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.934501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.934514 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.934803 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.935164 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.935176 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.935181 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.935355 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.935871 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.935877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.936155 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.936366 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.936657 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":72,"spinlock_wait_cycles":25600,"thread_start_us":117,"threads_started":1}
I20260928 14:41:17.937918 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.922738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:43331 (local address 127.1.201.61:46839)
0928 14:41:17.922878 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.922883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.923525 (+ 642us) server_negotiation.cc:407] Connection header received
0928 14:41:17.924303 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.924306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.924360 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.924426 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.925615 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.926408 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.931578 (+ 5170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.932001 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.932102 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.934287 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.934291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.934293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.934324 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.935924 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.936367 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.936370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.936371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.936452 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.936763 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.936766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.936767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.936958 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.937043 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.937632 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.937757 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:41:17.937515 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.922546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46839 (local address 127.1.201.2:43331)
0928 14:41:17.923400 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:17.923437 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:17.924128 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:17.924560 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:17.924566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:17.924972 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:17.925480 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.925492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.930509 (+ 5017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.930512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.931448 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:17.931459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.932154 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.932158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:17.932268 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.932748 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:17.932761 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:17.934124 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:17.936029 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.936032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.936034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.936252 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.936548 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:17.936551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:17.936553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:17.936649 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:17.937094 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:17.937097 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:17.937185 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 14:41:17.937271 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:41:17.937374 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":466,"spinlock_wait_cycles":64128,"thread_start_us":77,"threads_started":1}
I20260928 14:41:17.938719 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:17.922743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:53959 (local address 127.1.201.62:35941)
0928 14:41:17.922963 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:17.922967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:17.923518 (+ 551us) server_negotiation.cc:407] Connection header received
0928 14:41:17.924322 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:17.924326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:17.924374 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:17.924422 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:17.925639 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.926411 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.928159 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:17.928583 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:17.928708 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:17.932748 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:17.932754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:17.932760 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:17.932792 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:17.935580 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.936031 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.936035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.936036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.936118 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:17.936428 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:17.936431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:17.936434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:17.936630 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:17.936718 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:17.938441 (+ 1723us) server_negotiation.cc:299] Negotiation successful
0928 14:41:17.938558 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260928 14:41:17.939018 9052 heartbeater.cc:347] Connected to a master server at 127.1.201.60:44017
I20260928 14:41:17.939076 9053 heartbeater.cc:347] Connected to a master server at 127.1.201.61:46839
I20260928 14:41:17.939247 9052 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:17.939249 9053 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:17.939776 9053 heartbeater.cc:511] Master 127.1.201.61:46839 requested a full tablet report, sending...
I20260928 14:41:17.939776 9052 heartbeater.cc:511] Master 127.1.201.60:44017 requested a full tablet report, sending...
I20260928 14:41:17.940318 9051 heartbeater.cc:347] Connected to a master server at 127.1.201.62:35941
I20260928 14:41:17.940387 9051 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:17.940517 9051 heartbeater.cc:511] Master 127.1.201.62:35941 requested a full tablet report, sending...
I20260928 14:41:17.940814 8553 ts_manager.cc:194] Registered new tserver with Master: 14f5ba50d8e54f0490dc670239d05163 (127.1.201.2:39669)
I20260928 14:41:17.941110 7952 ts_manager.cc:194] Re-registered known tserver with Master: 14f5ba50d8e54f0490dc670239d05163 (127.1.201.2:39669)
I20260928 14:41:17.941617 7952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:53959
I20260928 14:41:17.941671 8738 ts_manager.cc:194] Registered new tserver with Master: 14f5ba50d8e54f0490dc670239d05163 (127.1.201.2:39669)
W20260928 14:41:18.039067 9058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:18.039326 9058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:18.039395 9058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:18.042974 9058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:18.043018 9058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:18.043066 9058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:18.047287 9058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab
--principal=oryx/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:37881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=37057
--tserver_master_addrs=127.1.201.62:35941,127.1.201.61:46839,127.1.201.60:44017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:18.048107 9058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:18.048925 9058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:18.057585 9064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:18.057611 9063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:18.057657 9058 server_base.cc:1034] running on GCE node
W20260928 14:41:18.057611 9066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:18.058185 9058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:18.058728 9058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:18.059892 9058 hybrid_clock.cc:648] HybridClock initialized: now 1790606478059872 us; error 28 us; skew 500 ppm
Sep 28 14:41:18 dist-test-slave-0n27 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606478, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:18.062587 9058 init.cc:377] Logged in from keytab as oryx/127.1.201.3@KRBTEST.COM (short username oryx)
I20260928 14:41:18.063289 9058 webserver.cc:460] Webserver started at http://127.1.201.3:37057/ using document root <none> and password file <none>
I20260928 14:41:18.063818 9058 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:18.064008 9058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:18.067488 9058 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260928 14:41:18.069528 9073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:18.070706 9058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:18.070837 9058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "e1bc95a0d3cd4b339026c59818e3fadf"
format_stamp: "Formatted at 2026-09-28 14:41:16 on dist-test-slave-0n27"
server_key: "a76dd4b7892bbb52358dc6c9d00aeb6a"
server_key_iv: "70be97b509ac96a7b12777e6d30009a4"
server_key_version: "encryptionkey@0"
I20260928 14:41:18.071290 9058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:18.088827 9058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:18.091570 9058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:18.091745 9058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:18.092275 9058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:18.093127 9058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:18.093174 9058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:18.093230 9058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:18.093261 9058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:18.102200 9058 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:37881
I20260928 14:41:18.102269 9186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:37881 every 8 connection(s)
I20260928 14:41:18.103425 9058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:41:18 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606478, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:18.113179 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 9058
Sep 28 14:41:18 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606478, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.60@KRBTEST.COM
Sep 28 14:41:18 dist-test-slave-0n27 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606478, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.61@KRBTEST.COM
I20260928 14:41:18.118388 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.105085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:48511 (local address 127.1.201.62:35941)
0928 14:41:18.105209 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.105215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.106706 (+ 1491us) server_negotiation.cc:407] Connection header received
0928 14:41:18.107370 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.107373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.107439 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.107491 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:18.108849 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.109563 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.110779 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.111011 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.111113 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.113461 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:18.113466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:18.113471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:18.113501 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:18.115239 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.116644 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.116647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.116649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.116739 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.117130 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.117135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.117137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.117350 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:18.117414 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.118054 (+ 640us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.118213 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:41:18.118541 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.105166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:36963 (local address 127.1.201.60:44017)
0928 14:41:18.105279 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.105283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.106250 (+ 967us) server_negotiation.cc:407] Connection header received
0928 14:41:18.107370 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.107373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.107439 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.107491 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:18.108722 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.109247 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.110805 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.111153 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.111222 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.114786 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:18.114789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:18.114792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:18.114821 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:18.116725 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.117263 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.117268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.117269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.117345 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.117654 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.117657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.117658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.117857 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:18.117933 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.118275 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.118382 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:41:18.119064 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.105367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:40495 (local address 127.1.201.61:46839)
0928 14:41:18.105486 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.105489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.106445 (+ 956us) server_negotiation.cc:407] Connection header received
0928 14:41:18.107342 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.107346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.107402 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.107512 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:18.109363 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.110170 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.111211 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.111499 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.111606 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.114428 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:18.114435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:18.114440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:18.114470 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:18.116700 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.117181 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.117186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.117187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.117256 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.117521 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.117524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.117526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.117745 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:18.117888 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.118870 (+ 982us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.118961 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:41:18.120826 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.105525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46839 (local address 127.1.201.3:40495)
0928 14:41:18.106372 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:18.106384 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:18.107160 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:18.107671 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:18.107677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:18.108046 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:18.108580 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.108595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.110286 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.110290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.111087 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.111096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.111747 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.111749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.111844 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.112230 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:18.112243 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:18.114306 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:18.116809 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:18.116812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:18.116814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:18.117088 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:18.117349 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:18.117352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:18.117354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:18.117429 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:18.117931 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:18.117937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:18.118031 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 14:41:18.118164 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 14:41:18.119613 (+ 1449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":101,"thread_start_us":201,"threads_started":1}
I20260928 14:41:18.120826 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.105525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:44017 (local address 127.1.201.3:36963)
0928 14:41:18.106102 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:18.106159 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:18.107163 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:18.107671 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:18.107678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:18.108051 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:18.108582 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.108596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.109381 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.109384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.110672 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.110682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.111352 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.111355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.111435 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.111919 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:18.111933 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:18.114614 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:18.116823 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:18.116825 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:18.116827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:18.117146 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:18.117447 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:18.117450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:18.117451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:18.117535 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:18.117986 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:18.117989 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:18.118079 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 14:41:18.118677 (+ 598us) client_negotiation.cc:240] Negotiation successful
0928 14:41:18.119751 (+ 1074us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":859,"spinlock_wait_cycles":13184,"thread_start_us":653,"threads_started":1}
I20260928 14:41:18.121564 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.105681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.3:48511)
0928 14:41:18.106605 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:18.106617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:18.107163 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:18.107671 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:18.107678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:18.108051 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:18.108747 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.108750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.109691 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.109695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.110668 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.110677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.111147 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.111151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.111295 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.111791 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:18.111813 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:18.113297 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:18.115783 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:18.115789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:18.115802 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:18.116171 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:18.116839 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:18.116844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:18.116847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:18.117007 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:18.117487 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:18.117494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:18.117760 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 14:41:18.118641 (+ 881us) client_negotiation.cc:240] Negotiation successful
0928 14:41:18.120846 (+ 2205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":1285,"spinlock_wait_cycles":151808,"thread_start_us":182,"threads_started":1}
I20260928 14:41:18.122124 9188 heartbeater.cc:347] Connected to a master server at 127.1.201.60:44017
I20260928 14:41:18.122157 9189 heartbeater.cc:347] Connected to a master server at 127.1.201.61:46839
I20260928 14:41:18.122328 9189 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:18.122329 9188 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:18.122876 9189 heartbeater.cc:511] Master 127.1.201.61:46839 requested a full tablet report, sending...
I20260928 14:41:18.122880 9188 heartbeater.cc:511] Master 127.1.201.60:44017 requested a full tablet report, sending...
I20260928 14:41:18.123509 9187 heartbeater.cc:347] Connected to a master server at 127.1.201.62:35941
I20260928 14:41:18.123588 9187 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:18.123729 9187 heartbeater.cc:511] Master 127.1.201.62:35941 requested a full tablet report, sending...
I20260928 14:41:18.124001 8738 ts_manager.cc:194] Registered new tserver with Master: e1bc95a0d3cd4b339026c59818e3fadf (127.1.201.3:37881)
I20260928 14:41:18.124285 7952 ts_manager.cc:194] Re-registered known tserver with Master: e1bc95a0d3cd4b339026c59818e3fadf (127.1.201.3:37881)
I20260928 14:41:18.124418 8553 ts_manager.cc:194] Registered new tserver with Master: e1bc95a0d3cd4b339026c59818e3fadf (127.1.201.3:37881)
I20260928 14:41:18.124975 7952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:48511
I20260928 14:41:18.126974 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.115141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.1.201.60:44017)
0928 14:41:18.115722 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.115728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.116498 (+ 770us) server_negotiation.cc:407] Connection header received
0928 14:41:18.116553 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.116556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.116603 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.116678 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:18.117579 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.118353 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.119791 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.120101 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.120203 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.122285 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:18.122288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:18.122290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:18.122312 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:18.125216 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.125667 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.125670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.125671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.125745 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.126039 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.126042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.126043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.126234 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:18.126342 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.126805 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.126881 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":90,"threads_started":1}
I20260928 14:41:18.130074 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.115913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.1.201.61:46839)
0928 14:41:18.119257 (+ 3344us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.119260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.119270 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:41:18.119297 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.119299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.119341 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.119430 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:18.120225 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.121163 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.124590 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.124880 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.124938 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.126568 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:18.126571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:18.126574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:18.126603 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:18.128076 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.128509 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.128513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.128514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.128602 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.128905 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.128909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.128910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.129094 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:18.129261 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.129517 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.129945 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3275,"thread_start_us":66,"threads_started":1}
I20260928 14:41:18.133539 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.115575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.1.201.62:35941)
0928 14:41:18.118566 (+ 2991us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.118570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.122627 (+ 4057us) server_negotiation.cc:407] Connection header received
0928 14:41:18.123182 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.123185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.123225 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.123269 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:18.127741 (+ 4472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.128230 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.129055 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.129334 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.129426 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.131016 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:18.131019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:18.131021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:18.131042 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:18.132066 (+ 1024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.132433 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.132436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.132437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.132511 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.132763 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.132765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.132767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.132934 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:18.133058 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.133281 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.133382 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2935,"thread_start_us":96,"threads_started":1}
I20260928 14:41:18.136142 7952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:41:18.139155 7952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:41:18.153831 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.149041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:33547 (local address 127.0.0.1:43174)
0928 14:41:18.149693 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:18.149726 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:18.149839 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:18.150495 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:18.150497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:18.150511 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:18.150574 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.150578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.151802 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.151806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.152680 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.152688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.153438 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.153441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.153518 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.153537 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:41:18.153594 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 14:41:18.153676 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":22,"thread_start_us":86,"threads_started":1}
I20260928 14:41:18.153877 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.149220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.1.201.1:33547)
0928 14:41:18.149626 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.149637 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.149816 (+ 179us) server_negotiation.cc:407] Connection header received
0928 14:41:18.150213 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.150218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.150377 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.150500 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:18.150686 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.151700 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.152805 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.153320 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.153437 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.153564 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.153645 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.153726 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":97,"threads_started":1}
I20260928 14:41:18.154731 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.149617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39669 (local address 127.0.0.1:38108)
0928 14:41:18.150679 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:18.150692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:18.150788 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:18.151137 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:18.151140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:18.151154 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:18.151237 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.151245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.152414 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.152417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.153301 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.153309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.154355 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.154359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.154443 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.154465 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:41:18.154547 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:41:18.154617 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":17,"thread_start_us":111,"threads_started":1}
I20260928 14:41:18.154799 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.149826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.1.201.2:39669)
0928 14:41:18.150114 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.150128 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.150783 (+ 655us) server_negotiation.cc:407] Connection header received
0928 14:41:18.150896 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.150899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.151025 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.151158 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:18.151323 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.152185 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.153409 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.154197 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.154350 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.154458 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.154575 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.154657 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260928 14:41:18.155864 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.149331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:37881 (local address 127.0.0.1:41978)
0928 14:41:18.150022 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:18.150035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:18.150110 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:18.151472 (+ 1362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:18.151474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:18.151489 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:18.151579 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.151584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.152759 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.152762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.154500 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.154509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.155485 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.155488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.155581 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.155620 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 14:41:18.155686 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:41:18.155747 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260928 14:41:18.156792 8848 tablet_service.cc:1467] Processing CreateTablet for tablet 81239731c28e41e78db361a19f6e08f8 (DEFAULT_TABLE table=test-table [id=2d0e909441fa48318dfede18bb3b9255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:18.156942 8985 tablet_service.cc:1467] Processing CreateTablet for tablet 81239731c28e41e78db361a19f6e08f8 (DEFAULT_TABLE table=test-table [id=2d0e909441fa48318dfede18bb3b9255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:18.157614 8848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81239731c28e41e78db361a19f6e08f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:18.157634 8985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81239731c28e41e78db361a19f6e08f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:18.159651 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.149977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.1.201.3:37881)
0928 14:41:18.151003 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.151012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.151029 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:41:18.151084 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.151088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.151245 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.151370 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:18.151699 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.152647 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.154627 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.155332 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.159275 (+ 3943us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.159409 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.159451 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.159522 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":72,"threads_started":1}
I20260928 14:41:18.162146 9121 tablet_service.cc:1467] Processing CreateTablet for tablet 81239731c28e41e78db361a19f6e08f8 (DEFAULT_TABLE table=test-table [id=2d0e909441fa48318dfede18bb3b9255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:18.162981 9121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81239731c28e41e78db361a19f6e08f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:18.165210 9216 tablet_bootstrap.cc:492] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: Bootstrap starting.
I20260928 14:41:18.165372 9215 tablet_bootstrap.cc:492] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: Bootstrap starting.
I20260928 14:41:18.167023 9216 tablet_bootstrap.cc:654] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:18.167335 9215 tablet_bootstrap.cc:654] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:18.167728 9216 log.cc:824] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:18.168121 9215 log.cc:824] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:18.169827 9219 tablet_bootstrap.cc:492] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: Bootstrap starting.
I20260928 14:41:18.169953 9216 tablet_bootstrap.cc:492] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: No bootstrap required, opened a new log
I20260928 14:41:18.170173 9216 ts_tablet_manager.cc:1397] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:41:18.171022 9215 tablet_bootstrap.cc:492] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: No bootstrap required, opened a new log
I20260928 14:41:18.171139 9215 ts_tablet_manager.cc:1397] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 14:41:18.172075 9219 tablet_bootstrap.cc:654] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:18.172825 9219 log.cc:824] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:18.174414 9216 raft_consensus.cc:348] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } } peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } }
I20260928 14:41:18.174647 9216 raft_consensus.cc:374] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:18.174692 9216 raft_consensus.cc:729] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd640aa359054a81aa2e5011e31b16ec, State: Initialized, Role: FOLLOWER
I20260928 14:41:18.174703 9219 tablet_bootstrap.cc:492] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: No bootstrap required, opened a new log
I20260928 14:41:18.174468 9215 raft_consensus.cc:348] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } } peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } }
I20260928 14:41:18.174793 9215 raft_consensus.cc:374] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:18.174849 9215 raft_consensus.cc:729] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f5ba50d8e54f0490dc670239d05163, State: Initialized, Role: FOLLOWER
I20260928 14:41:18.174845 9219 ts_tablet_manager.cc:1397] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:41:18.175087 9216 consensus_queue.cc:260] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } } peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } }
I20260928 14:41:18.175237 9215 consensus_queue.cc:260] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } } peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } }
I20260928 14:41:18.175766 8914 heartbeater.cc:503] Master 127.1.201.62:35941 was elected leader, sending a full tablet report...
I20260928 14:41:18.175962 9216 ts_tablet_manager.cc:1428] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:41:18.176055 9215 ts_tablet_manager.cc:1428] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:41:18.176321 9051 heartbeater.cc:503] Master 127.1.201.62:35941 was elected leader, sending a full tablet report...
I20260928 14:41:18.182472 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.176889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:44017 (local address 127.0.0.1:56858)
0928 14:41:18.177357 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:18.177371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:18.177437 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:18.178413 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:18.178416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:18.178432 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:18.178506 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.178511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.180402 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.180404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.181223 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.181232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.182090 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.182093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.182197 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.182217 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:41:18.182295 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:41:18.182361 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394}
I20260928 14:41:18.182472 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.177883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.1.201.60:44017)
0928 14:41:18.178087 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.178091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.178110 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:41:18.178169 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.178171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.178221 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.178308 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:18.179565 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.180313 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.181347 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.181974 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.182090 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.182225 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.182306 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.182372 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260928 14:41:18.182327 9219 raft_consensus.cc:348] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } } peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } }
I20260928 14:41:18.182785 9219 raft_consensus.cc:374] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:18.182852 9219 raft_consensus.cc:729] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bc95a0d3cd4b339026c59818e3fadf, State: Initialized, Role: FOLLOWER
I20260928 14:41:18.183197 8744 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 14:41:18.183230 9219 consensus_queue.cc:260] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } } peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } }
I20260928 14:41:18.183831 9187 heartbeater.cc:503] Master 127.1.201.62:35941 was elected leader, sending a full tablet report...
I20260928 14:41:18.184041 9219 ts_tablet_manager.cc:1428] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260928 14:41:18.185873 8778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5402e44e06f74daeb4a70d506780619a. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:18.185963 8778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: This master's current role is: LEARNER
W20260928 14:41:18.240132 8917 tablet.cc:2369] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:41:18.354583 9190 tablet.cc:2369] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:41:18.421957 9054 tablet.cc:2369] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:41:18.428906 9221 raft_consensus.cc:482] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:41:18.429113 9221 raft_consensus.cc:504] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } } peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } }
I20260928 14:41:18.430362 9221 leader_election.cc:288] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bc95a0d3cd4b339026c59818e3fadf (127.1.201.3:37881), 14f5ba50d8e54f0490dc670239d05163 (127.1.201.2:39669)
I20260928 14:41:18.433887 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.430705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:33585 (local address 127.1.201.3:37881)
0928 14:41:18.430823 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.430828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.430887 (+ 59us) server_negotiation.cc:407] Connection header received
0928 14:41:18.431066 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.431069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.431128 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.431211 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:18.431470 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.431970 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.432969 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.433356 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.433468 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.433526 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.433689 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.433755 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:41:18.433912 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.430575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:37881 (local address 127.1.201.1:33585)
0928 14:41:18.430814 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:18.430829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:18.430938 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:18.431223 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:18.431226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:18.431254 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:18.431359 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.431363 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.432074 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.432079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.432834 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.432842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.433470 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.433473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.433556 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.433577 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:41:18.433669 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:41:18.433739 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"spinlock_wait_cycles":4480,"thread_start_us":70,"threads_started":1}
I20260928 14:41:18.434051 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.431044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:60215 (local address 127.1.201.2:39669)
0928 14:41:18.431157 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.431163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.431219 (+ 56us) server_negotiation.cc:407] Connection header received
0928 14:41:18.431342 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.431347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.431389 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.431470 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:18.431718 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.432175 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.433146 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.433510 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.433619 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.433672 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.433889 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.433948 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:41:18.434072 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.430946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39669 (local address 127.1.201.1:60215)
0928 14:41:18.431158 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:18.431169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:18.431262 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:18.431479 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:18.431482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:18.431498 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:18.431596 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.431602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.432280 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.432284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.433026 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:18.433034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.433622 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.433626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:18.433721 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.433742 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:41:18.433871 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 14:41:18.433938 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"spinlock_wait_cycles":1408,"thread_start_us":83,"threads_started":1}
I20260928 14:41:18.434643 9005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81239731c28e41e78db361a19f6e08f8" candidate_uuid: "bd640aa359054a81aa2e5011e31b16ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f5ba50d8e54f0490dc670239d05163" is_pre_election: true
I20260928 14:41:18.434643 9141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81239731c28e41e78db361a19f6e08f8" candidate_uuid: "bd640aa359054a81aa2e5011e31b16ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" is_pre_election: true
I20260928 14:41:18.434907 9141 raft_consensus.cc:2454] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd640aa359054a81aa2e5011e31b16ec in term 0.
I20260928 14:41:18.434907 9005 raft_consensus.cc:2454] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd640aa359054a81aa2e5011e31b16ec in term 0.
I20260928 14:41:18.435333 8802 leader_election.cc:302] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd640aa359054a81aa2e5011e31b16ec, e1bc95a0d3cd4b339026c59818e3fadf; no voters:
I20260928 14:41:18.435614 9221 raft_consensus.cc:2781] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:41:18.435745 9221 raft_consensus.cc:482] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:41:18.435806 9221 raft_consensus.cc:3037] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:18.436772 9221 raft_consensus.cc:504] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } } peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } }
I20260928 14:41:18.437116 9221 leader_election.cc:288] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [CANDIDATE]: Term 1 election: Requested vote from peers e1bc95a0d3cd4b339026c59818e3fadf (127.1.201.3:37881), 14f5ba50d8e54f0490dc670239d05163 (127.1.201.2:39669)
I20260928 14:41:18.437441 9141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81239731c28e41e78db361a19f6e08f8" candidate_uuid: "bd640aa359054a81aa2e5011e31b16ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bc95a0d3cd4b339026c59818e3fadf"
I20260928 14:41:18.437463 9005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81239731c28e41e78db361a19f6e08f8" candidate_uuid: "bd640aa359054a81aa2e5011e31b16ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f5ba50d8e54f0490dc670239d05163"
I20260928 14:41:18.437580 9005 raft_consensus.cc:3037] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:18.437572 9141 raft_consensus.cc:3037] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:18.438894 9005 raft_consensus.cc:2454] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd640aa359054a81aa2e5011e31b16ec in term 1.
I20260928 14:41:18.438882 9141 raft_consensus.cc:2454] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd640aa359054a81aa2e5011e31b16ec in term 1.
I20260928 14:41:18.439153 8802 leader_election.cc:302] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd640aa359054a81aa2e5011e31b16ec, e1bc95a0d3cd4b339026c59818e3fadf; no voters:
I20260928 14:41:18.439333 9221 raft_consensus.cc:2781] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:18.439482 9221 raft_consensus.cc:686] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 1 LEADER]: Becoming Leader. State: Replica: bd640aa359054a81aa2e5011e31b16ec, State: Running, Role: LEADER
I20260928 14:41:18.439786 9221 consensus_queue.cc:237] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } } peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } }
I20260928 14:41:18.442870 7951 catalog_manager.cc:5495] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec reported cstate change: term changed from 0 to 1, leader changed from <none> to bd640aa359054a81aa2e5011e31b16ec (127.1.201.1). New cstate: current_term: 1 leader_uuid: "bd640aa359054a81aa2e5011e31b16ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } health_report { overall_health: UNKNOWN } } }
I20260928 14:41:18.444480 8511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } } }
I20260928 14:41:18.444613 8511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:18.447368 9232 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: attempting to promote NON_VOTER 999ad913e9c04ad581d6db34d5705d82 to VOTER
I20260928 14:41:18.447428 9205 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: attempting to promote NON_VOTER 999ad913e9c04ad581d6db34d5705d82 to VOTER
I20260928 14:41:18.447711 9232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } }
W20260928 14:41:18.448282 9205 raft_consensus.cc:1053] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: Unable to promote non-voter 999ad913e9c04ad581d6db34d5705d82: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: NON_VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: true } }.
Pending config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } }
I20260928 14:41:18.448693 8744 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 LEARNER]: Refusing update from remote peer 5402e44e06f74daeb4a70d506780619a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 14:41:18.448801 8563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 FOLLOWER]: Refusing update from remote peer 5402e44e06f74daeb4a70d506780619a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 14:41:18.449024 9232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Connected to new peer: Peer: permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 14:41:18.449203 8511 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 14:41:18.450690 9205 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 999ad913e9c04ad581d6db34d5705d82 (127.1.201.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:18.451140 8744 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 999ad913e9c04ad581d6db34d5705d82 (127.1.201.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:18.451920 9205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:18.452020 9205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:18.452155 9205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:18.452242 9205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:18.452204 8511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:18.452278 8511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:18.452550 8778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:18.452661 8778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:41:18.452880 8563 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 999ad913e9c04ad581d6db34d5705d82 (127.1.201.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:18.454036 9235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5402e44e06f74daeb4a70d506780619a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:18.454162 9235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:41:18.515463 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.512415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.1.201.1:33547)
0928 14:41:18.512517 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.512520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.512531 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:41:18.512588 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.512590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.512638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.512788 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:18.512979 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.513467 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.514326 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.514541 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.514618 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.514847 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:18.515212 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:18.515288 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.515338 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.515377 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:41:18.520829 9141 raft_consensus.cc:1261] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf [term 1 FOLLOWER]: Refusing update from remote peer bd640aa359054a81aa2e5011e31b16ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:18.520835 9005 raft_consensus.cc:1261] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163 [term 1 FOLLOWER]: Refusing update from remote peer bd640aa359054a81aa2e5011e31b16ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:18.521526 9221 consensus_queue.cc:1035] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:18.521727 9230 consensus_queue.cc:1035] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:18.530416 9239 mvcc.cc:205] Tried to move back new op lower bound from 7334324136014041088 to 7334324135691632640. Current Snapshot: MvccSnapshot[applied={T|T < 7334324136014041088}]
I20260928 14:41:18.538820 7951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:18.538993 7951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:18.542240 7951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5402e44e06f74daeb4a70d506780619a: Sending DeleteTablet for 3 replicas of tablet 81239731c28e41e78db361a19f6e08f8
I20260928 14:41:18.543090 9121 tablet_service.cc:1514] Processing DeleteTablet for tablet 81239731c28e41e78db361a19f6e08f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:18 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:41978
I20260928 14:41:18.543581 8848 tablet_service.cc:1514] Processing DeleteTablet for tablet 81239731c28e41e78db361a19f6e08f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:18 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:43174
I20260928 14:41:18.544032 9248 tablet_replica.cc:330] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: stopping tablet replica
I20260928 14:41:18.544312 9248 raft_consensus.cc:2229] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:41:18.544349 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 7920
I20260928 14:41:18.544358 8985 tablet_service.cc:1514] Processing DeleteTablet for tablet 81239731c28e41e78db361a19f6e08f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:18 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:38108
I20260928 14:41:18.546012 9248 raft_consensus.cc:2258] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:18.547307 9248 ts_tablet_manager.cc:1905] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:18.550675 9248 ts_tablet_manager.cc:1918] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:18.550771 9248 log.cc:1196] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/81239731c28e41e78db361a19f6e08f8
I20260928 14:41:18.551075 9248 ts_tablet_manager.cc:1939] T 81239731c28e41e78db361a19f6e08f8 P bd640aa359054a81aa2e5011e31b16ec: Deleting consensus metadata
W20260928 14:41:18.552320 8539 connection.cc:535] server connection from 127.0.0.1:57204 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 14:41:18.552489 8802 connection.cc:535] server connection from 127.0.0.1:43174 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:41:18.553341 9249 tablet_replica.cc:330] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: stopping tablet replica
I20260928 14:41:18.553601 9249 raft_consensus.cc:2229] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:18.553926 9249 raft_consensus.cc:2258] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:18.554234 9246 tablet_replica.cc:330] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: stopping tablet replica
I20260928 14:41:18.554463 9246 raft_consensus.cc:2229] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:18.554700 9246 raft_consensus.cc:2258] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:18.554780 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.545024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.1.201.61:46839)
0928 14:41:18.545181 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.545186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.545549 (+ 363us) server_negotiation.cc:407] Connection header received
0928 14:41:18.545732 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.545735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.545801 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.545852 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:18.546837 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.547590 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.548848 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.549184 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.549253 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.551157 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:18.551161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:18.551164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:18.551188 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:18.552849 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.553359 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.553363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.553365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.553438 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.553849 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.553852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.553853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.554041 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:18.554129 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.554529 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.554641 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:41:18.555898 9249 ts_tablet_manager.cc:1905] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:18.557708 9249 ts_tablet_manager.cc:1918] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:18.557864 9249 log.cc:1196] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/81239731c28e41e78db361a19f6e08f8
I20260928 14:41:18.559238 9246 ts_tablet_manager.cc:1905] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:18.560822 9249 ts_tablet_manager.cc:1939] T 81239731c28e41e78db361a19f6e08f8 P 14f5ba50d8e54f0490dc670239d05163: Deleting consensus metadata
I20260928 14:41:18.563053 9246 ts_tablet_manager.cc:1918] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:18.563151 9246 log.cc:1196] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/81239731c28e41e78db361a19f6e08f8
I20260928 14:41:18.563515 9246 ts_tablet_manager.cc:1939] T 81239731c28e41e78db361a19f6e08f8 P e1bc95a0d3cd4b339026c59818e3fadf: Deleting consensus metadata
W20260928 14:41:18.564527 9075 connection.cc:441] server connection from 127.0.0.1:41978 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41978 (request call id 1) could send its response
W20260928 14:41:18.564750 8939 connection.cc:441] server connection from 127.0.0.1:38108 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38108 (request call id 1) could send its response
I20260928 14:41:18.566150 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.544035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.1.201.60:44017)
0928 14:41:18.544135 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.544139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.556365 (+ 12226us) server_negotiation.cc:407] Connection header received
0928 14:41:18.556504 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.556507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.556547 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.556589 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:18.557653 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.558452 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.559667 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.560048 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.560130 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.562065 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:18.562069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:18.562071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:18.562097 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:18.563895 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.564806 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.564809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.564811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.564889 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.565176 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.565183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.565186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.565390 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:18.565511 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.565863 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.565980 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":38}
I20260928 14:41:18.567144 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.558233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.1.201.60:44017)
0928 14:41:18.558341 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.558345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.558433 (+ 88us) server_negotiation.cc:407] Connection header received
0928 14:41:18.558704 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.558708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.558782 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.558834 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:18.559843 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.560555 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.561802 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.562097 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.562173 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.563892 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:18.563895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:18.563898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:18.563920 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:18.565422 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.565921 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.565924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.565925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.565995 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.566313 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.566318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.566323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.566519 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:18.566611 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.566943 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.567048 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":42}
I20260928 14:41:18.572736 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:18.560452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.1.201.61:46839)
0928 14:41:18.560563 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:18.560566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:18.565331 (+ 4765us) server_negotiation.cc:407] Connection header received
0928 14:41:18.565467 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:18.565470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:18.565506 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:18.565547 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:18.566348 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.567054 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.568262 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:18.568463 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:18.568541 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:18.569981 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:18.569984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:18.569985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:18.570005 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:18.571329 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.571767 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.571769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.571770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.571832 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:18.572072 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:18.572073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:18.572074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:18.572207 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:18.572290 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:18.572518 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 14:41:18.572594 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:41:19.532083 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:19.531722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.3:42525)
0928 14:41:19.531926 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:19.531996 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":61,"threads_started":1}
W20260928 14:41:19.532377 9187 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:35941 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
I20260928 14:41:19.534683 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:19.534276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.1:52317)
0928 14:41:19.534501 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:19.534596 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
W20260928 14:41:19.535161 8914 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:35941 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
I20260928 14:41:19.539569 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:19.539234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.2:37797)
0928 14:41:19.539424 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:19.539500 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":57,"threads_started":1}
W20260928 14:41:19.539876 9051 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:35941 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
I20260928 14:41:20.043784 9259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5402e44e06f74daeb4a70d506780619a)
I20260928 14:41:20.043957 9259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } }
I20260928 14:41:20.044864 9259 leader_election.cc:288] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5402e44e06f74daeb4a70d506780619a (127.1.201.62:35941), 0511389fb9e14e169f05b776b7e40d5d (127.1.201.61:46839)
I20260928 14:41:20.045373 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.044961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56656)
0928 14:41:20.045221 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.045300 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
W20260928 14:41:20.045552 8719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
I20260928 14:41:20.046422 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.046179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56664)
0928 14:41:20.046303 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.046354 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:41:20.046530 9264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5402e44e06f74daeb4a70d506780619a)
W20260928 14:41:20.046551 8719 leader_election.cc:334] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5402e44e06f74daeb4a70d506780619a (127.1.201.62:35941): Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
I20260928 14:41:20.046691 9264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } }
I20260928 14:41:20.047501 9264 leader_election.cc:288] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5402e44e06f74daeb4a70d506780619a (127.1.201.62:35941), 999ad913e9c04ad581d6db34d5705d82 (127.1.201.60:44017)
I20260928 14:41:20.047971 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.047566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56668)
0928 14:41:20.047808 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.047876 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":61,"threads_started":1}
W20260928 14:41:20.048111 8537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
I20260928 14:41:20.048933 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.048723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56680)
0928 14:41:20.048823 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.048876 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 14:41:20.049041 8537 leader_election.cc:334] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5402e44e06f74daeb4a70d506780619a (127.1.201.62:35941): Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
I20260928 14:41:20.049577 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.045067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46839 (local address 127.0.0.1:57240)
0928 14:41:20.045283 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.045298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:20.045390 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:20.045677 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:20.045679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:20.045694 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:20.045859 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.045868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.046969 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.046972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.047970 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.047980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.048896 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.048900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.049253 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.049279 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:41:20.049374 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:41:20.049443 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":221,"thread_start_us":62,"threads_started":1}
I20260928 14:41:20.049606 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.045196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.1.201.61:46839)
0928 14:41:20.045454 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:20.045460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:20.045473 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:41:20.045519 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:20.045521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:20.045553 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:20.045653 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:20.045970 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.046850 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.048152 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.048786 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.048879 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.048935 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:20.049411 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 14:41:20.049525 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":185,"thread_start_us":63,"threads_started":1}
I20260928 14:41:20.050038 8563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "999ad913e9c04ad581d6db34d5705d82" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0511389fb9e14e169f05b776b7e40d5d" is_pre_election: true
I20260928 14:41:20.050202 8563 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 999ad913e9c04ad581d6db34d5705d82 in term 1.
I20260928 14:41:20.050480 8720 leader_election.cc:302] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0511389fb9e14e169f05b776b7e40d5d, 999ad913e9c04ad581d6db34d5705d82; no voters: 5402e44e06f74daeb4a70d506780619a
I20260928 14:41:20.050695 9259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:41:20.050752 9259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5402e44e06f74daeb4a70d506780619a)
I20260928 14:41:20.050788 9259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:41:20.051569 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.047703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:44017 (local address 127.0.0.1:56902)
0928 14:41:20.047918 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.047931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:20.048011 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:20.048304 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:20.048307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:20.048325 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:20.048399 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.048404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.049361 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.049367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.050479 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.050489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.051306 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.051309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.051370 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.051387 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 14:41:20.051445 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:41:20.051489 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":55,"threads_started":1}
I20260928 14:41:20.051637 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.047817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.1.201.60:44017)
0928 14:41:20.048036 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:20.048045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:20.048104 (+ 59us) server_negotiation.cc:407] Connection header received
0928 14:41:20.048146 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:20.048148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:20.048191 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:20.048299 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:20.048517 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.049263 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.050603 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.051180 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.051294 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.051342 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:20.051476 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 14:41:20.051533 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260928 14:41:20.051980 8744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0511389fb9e14e169f05b776b7e40d5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "999ad913e9c04ad581d6db34d5705d82" is_pre_election: true
I20260928 14:41:20.052083 9259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } }
I20260928 14:41:20.052239 8744 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0511389fb9e14e169f05b776b7e40d5d in current term 2: Already voted for candidate 999ad913e9c04ad581d6db34d5705d82 in this term.
I20260928 14:41:20.052409 9259 leader_election.cc:288] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [CANDIDATE]: Term 2 election: Requested vote from peers 5402e44e06f74daeb4a70d506780619a (127.1.201.62:35941), 0511389fb9e14e169f05b776b7e40d5d (127.1.201.61:46839)
I20260928 14:41:20.052599 8539 leader_election.cc:302] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0511389fb9e14e169f05b776b7e40d5d; no voters: 5402e44e06f74daeb4a70d506780619a, 999ad913e9c04ad581d6db34d5705d82
I20260928 14:41:20.052750 8563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "999ad913e9c04ad581d6db34d5705d82" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0511389fb9e14e169f05b776b7e40d5d"
I20260928 14:41:20.052912 9264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:41:20.053130 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.052500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56686)
0928 14:41:20.053043 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.053089 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486}
I20260928 14:41:20.053695 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.053509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56702)
0928 14:41:20.053599 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.053640 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 14:41:20.053896 9264 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260928 14:41:20.053889 8719 leader_election.cc:334] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5402e44e06f74daeb4a70d506780619a (127.1.201.62:35941): Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
I20260928 14:41:20.054739 8563 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 999ad913e9c04ad581d6db34d5705d82 in term 2.
I20260928 14:41:20.055298 8720 leader_election.cc:302] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0511389fb9e14e169f05b776b7e40d5d, 999ad913e9c04ad581d6db34d5705d82; no voters: 5402e44e06f74daeb4a70d506780619a
I20260928 14:41:20.055480 9259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:41:20.055707 9259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [term 2 LEADER]: Becoming Leader. State: Replica: 999ad913e9c04ad581d6db34d5705d82, State: Running, Role: LEADER
I20260928 14:41:20.055938 9259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } }
I20260928 14:41:20.056843 9269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999ad913e9c04ad581d6db34d5705d82. Latest consensus state: current_term: 2 leader_uuid: "999ad913e9c04ad581d6db34d5705d82" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:20.056974 9269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:20.057227 9271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:20.059295 9271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:20.059687 9271 catalog_manager.cc:1259] Loaded cluster ID: 01827db434a249b786859d1c0e3c185d
I20260928 14:41:20.059751 9271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:20.060217 9271 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:20.060643 9271 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Loaded TSK: 0
I20260928 14:41:20.061301 9271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:41:20.076924 8738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:41:20.079742 8738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:41:20.082221 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.082021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56732)
0928 14:41:20.082128 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.082173 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 14:41:20.082248 8563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [term 2 FOLLOWER]: Refusing update from remote peer 999ad913e9c04ad581d6db34d5705d82: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 14:41:20.082686 9259 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 14:41:20.082789 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.082566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.0.0.1:56736)
0928 14:41:20.082692 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.082739 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:41:20.082892 8719 consensus_peers.cc:487] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 -> Peer 5402e44e06f74daeb4a70d506780619a (127.1.201.62:35941): Couldn't send request to peer 5402e44e06f74daeb4a70d506780619a. Status: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:41:20.083950 9264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999ad913e9c04ad581d6db34d5705d82. Latest consensus state: current_term: 2 leader_uuid: "999ad913e9c04ad581d6db34d5705d82" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:20.084061 9264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:41:20.085451 9264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "999ad913e9c04ad581d6db34d5705d82" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:20.085568 9264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0511389fb9e14e169f05b776b7e40d5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:41:20.085826 9269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "999ad913e9c04ad581d6db34d5705d82" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5402e44e06f74daeb4a70d506780619a" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35941 } } peers { permanent_uuid: "0511389fb9e14e169f05b776b7e40d5d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46839 } attrs { promote: false } } peers { permanent_uuid: "999ad913e9c04ad581d6db34d5705d82" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 44017 } attrs { promote: false } } }
I20260928 14:41:20.085939 9269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:20.094789 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.090081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39669 (local address 127.0.0.1:38118)
0928 14:41:20.090152 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.090166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:20.090239 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:20.091563 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:20.091565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:20.091578 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:20.091656 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.091662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.092612 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.092618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.093480 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.093488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.094343 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.094347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.094502 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.094524 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:41:20.094577 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:41:20.094637 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":53}
I20260928 14:41:20.094936 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.090553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.1.201.1:33547)
0928 14:41:20.091013 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:20.091019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:20.091033 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:41:20.091100 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:20.091105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:20.091324 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:20.091442 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:20.091688 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.092358 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.093646 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.094139 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.094269 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.094328 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:20.094706 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 14:41:20.094803 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":68,"threads_started":1}
I20260928 14:41:20.094949 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.090320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:33547 (local address 127.0.0.1:43190)
0928 14:41:20.090404 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.090418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:20.090495 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:20.091457 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:20.091460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:20.091476 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:20.091562 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.091568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.092456 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.092461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.093509 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.093518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.094267 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.094271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.094374 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.094446 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 14:41:20.094515 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:41:20.094584 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":35}
I20260928 14:41:20.095402 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.090176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.1.201.2:39669)
0928 14:41:20.091230 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:20.091234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:20.091339 (+ 105us) server_negotiation.cc:407] Connection header received
0928 14:41:20.091387 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:20.091390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:20.091435 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:20.091557 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:20.091758 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.092496 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.093589 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.094218 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.095083 (+ 865us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.095173 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:20.095208 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 14:41:20.095270 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":60,"threads_started":1}
I20260928 14:41:20.095431 8848 tablet_service.cc:1467] Processing CreateTablet for tablet 994a211e2ff64fed86b85b643538045e (DEFAULT_TABLE table=test-table [id=2696a4c7fd7c483dad279b19e44c6e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:20.095715 8848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994a211e2ff64fed86b85b643538045e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:20.096004 8985 tablet_service.cc:1467] Processing CreateTablet for tablet 994a211e2ff64fed86b85b643538045e (DEFAULT_TABLE table=test-table [id=2696a4c7fd7c483dad279b19e44c6e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:20.096239 8985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994a211e2ff64fed86b85b643538045e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:20.096328 9280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.090624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:37881 (local address 127.0.0.1:41982)
0928 14:41:20.090832 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.090858 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:20.090957 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:20.091687 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:20.091690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:20.091708 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:20.091788 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.091793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.092906 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.092909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.094670 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.094678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.095956 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.095960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.096060 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.096079 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:41:20.096157 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:41:20.096218 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260928 14:41:20.096390 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.090729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.1.201.3:37881)
0928 14:41:20.091315 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:20.091320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:20.091336 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:41:20.091387 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:20.091390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:20.091435 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:20.091582 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:20.091904 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.092790 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.094776 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.095732 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.095831 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.095887 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:20.096197 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 14:41:20.096279 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260928 14:41:20.097870 9282 tablet_bootstrap.cc:492] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec: Bootstrap starting.
I20260928 14:41:20.098232 9121 tablet_service.cc:1467] Processing CreateTablet for tablet 994a211e2ff64fed86b85b643538045e (DEFAULT_TABLE table=test-table [id=2696a4c7fd7c483dad279b19e44c6e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:20.098346 9283 tablet_bootstrap.cc:492] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163: Bootstrap starting.
I20260928 14:41:20.098477 9121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994a211e2ff64fed86b85b643538045e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:20.099155 9283 tablet_bootstrap.cc:654] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:20.100297 9283 tablet_bootstrap.cc:492] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163: No bootstrap required, opened a new log
I20260928 14:41:20.100380 9283 ts_tablet_manager.cc:1397] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:20.100819 9283 raft_consensus.cc:348] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } } peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } }
I20260928 14:41:20.100894 9285 tablet_bootstrap.cc:492] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf: Bootstrap starting.
I20260928 14:41:20.100937 9283 raft_consensus.cc:374] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:20.100975 9283 raft_consensus.cc:729] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f5ba50d8e54f0490dc670239d05163, State: Initialized, Role: FOLLOWER
I20260928 14:41:20.101104 9283 consensus_queue.cc:260] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } } peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } }
I20260928 14:41:20.101441 9283 ts_tablet_manager.cc:1428] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:20.101801 9282 tablet_bootstrap.cc:654] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:20.101886 9285 tablet_bootstrap.cc:654] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:20.102412 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.101866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.2:32805)
0928 14:41:20.102243 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.102305 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":96,"threads_started":1}
I20260928 14:41:20.103144 9285 tablet_bootstrap.cc:492] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf: No bootstrap required, opened a new log
I20260928 14:41:20.103245 9285 ts_tablet_manager.cc:1397] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:41:20.103673 9285 raft_consensus.cc:348] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } } peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } }
I20260928 14:41:20.103777 9285 raft_consensus.cc:374] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:20.103809 9285 raft_consensus.cc:729] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bc95a0d3cd4b339026c59818e3fadf, State: Initialized, Role: FOLLOWER
I20260928 14:41:20.103924 9285 consensus_queue.cc:260] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } } peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } }
I20260928 14:41:20.104211 9285 ts_tablet_manager.cc:1428] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:41:20.104434 9052 heartbeater.cc:503] Master 127.1.201.60:44017 was elected leader, sending a full tablet report...
I20260928 14:41:20.104892 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.104315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.2:41055)
0928 14:41:20.104790 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.104841 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417}
I20260928 14:41:20.106719 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.104531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.3:53229)
0928 14:41:20.106584 (+ 2053us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.106635 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1996,"thread_start_us":83,"threads_started":1}
I20260928 14:41:20.107385 9188 heartbeater.cc:503] Master 127.1.201.60:44017 was elected leader, sending a full tablet report...
I20260928 14:41:20.107468 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.107259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.3:51819)
0928 14:41:20.107378 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.107419 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 14:41:20.108631 9282 tablet_bootstrap.cc:492] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec: No bootstrap required, opened a new log
I20260928 14:41:20.108741 9282 ts_tablet_manager.cc:1397] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260928 14:41:20.109062 9282 raft_consensus.cc:348] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } } peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } }
I20260928 14:41:20.109195 9282 raft_consensus.cc:374] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:20.109238 9282 raft_consensus.cc:729] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd640aa359054a81aa2e5011e31b16ec, State: Initialized, Role: FOLLOWER
I20260928 14:41:20.109365 9282 consensus_queue.cc:260] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } } peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } }
I20260928 14:41:20.109727 9282 ts_tablet_manager.cc:1428] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:20.110389 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.110045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.1:35821)
0928 14:41:20.110264 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.110313 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
I20260928 14:41:20.111167 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.110926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.1:35463)
0928 14:41:20.111067 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.111112 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 14:41:20.111411 8915 heartbeater.cc:503] Master 127.1.201.60:44017 was elected leader, sending a full tablet report...
I20260928 14:41:20.435307 9286 raft_consensus.cc:482] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:41:20.435460 9286 raft_consensus.cc:504] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } } peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } }
I20260928 14:41:20.436529 9286 leader_election.cc:288] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd640aa359054a81aa2e5011e31b16ec (127.1.201.1:33547), e1bc95a0d3cd4b339026c59818e3fadf (127.1.201.3:37881)
I20260928 14:41:20.440497 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.436868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:56179 (local address 127.1.201.1:33547)
0928 14:41:20.437007 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:20.437011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:20.437028 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:41:20.437148 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:20.437151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:20.437205 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:20.437276 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:20.437649 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.438284 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.439590 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.439978 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.440095 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.440158 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:20.440330 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:41:20.440397 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:41:20.440498 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.436761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:33547 (local address 127.1.201.2:56179)
0928 14:41:20.436900 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.436916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:20.437024 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:20.437305 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:20.437309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:20.437329 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:20.437472 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.437479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.438403 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.438406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.439485 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.439494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.440106 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.440109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.440207 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.440234 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:41:20.440316 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:41:20.440376 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"spinlock_wait_cycles":2304}
I20260928 14:41:20.440569 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.437139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:56267 (local address 127.1.201.3:37881)
0928 14:41:20.437291 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:20.437295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:20.437324 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:41:20.437464 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:20.437467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:20.437527 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:20.437630 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:20.437900 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.438591 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.439668 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.440033 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.440140 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.440197 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:20.440392 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 14:41:20.440454 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:41:20.440969 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.436998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:37881 (local address 127.1.201.2:56267)
0928 14:41:20.437264 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.437276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:20.437366 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:20.437649 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:20.437652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:20.437669 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:20.437773 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.437800 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.438722 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.438724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.439542 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:20.439554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.440145 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.440148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:20.440273 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.440295 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:41:20.440377 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:41:20.440440 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":4,"spinlock_wait_cycles":384,"thread_start_us":113,"threads_started":1}
I20260928 14:41:20.441031 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994a211e2ff64fed86b85b643538045e" candidate_uuid: "14f5ba50d8e54f0490dc670239d05163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd640aa359054a81aa2e5011e31b16ec" is_pre_election: true
I20260928 14:41:20.441258 8868 raft_consensus.cc:2454] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14f5ba50d8e54f0490dc670239d05163 in term 0.
I20260928 14:41:20.441393 9141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994a211e2ff64fed86b85b643538045e" candidate_uuid: "14f5ba50d8e54f0490dc670239d05163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" is_pre_election: true
I20260928 14:41:20.441517 9141 raft_consensus.cc:2454] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14f5ba50d8e54f0490dc670239d05163 in term 0.
I20260928 14:41:20.441701 8939 leader_election.cc:302] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f5ba50d8e54f0490dc670239d05163, bd640aa359054a81aa2e5011e31b16ec; no voters:
I20260928 14:41:20.441951 9286 raft_consensus.cc:2781] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:41:20.441999 9286 raft_consensus.cc:482] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:41:20.442022 9286 raft_consensus.cc:3037] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:20.443032 9286 raft_consensus.cc:504] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } } peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } }
I20260928 14:41:20.443349 9286 leader_election.cc:288] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [CANDIDATE]: Term 1 election: Requested vote from peers bd640aa359054a81aa2e5011e31b16ec (127.1.201.1:33547), e1bc95a0d3cd4b339026c59818e3fadf (127.1.201.3:37881)
I20260928 14:41:20.443665 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994a211e2ff64fed86b85b643538045e" candidate_uuid: "14f5ba50d8e54f0490dc670239d05163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd640aa359054a81aa2e5011e31b16ec"
I20260928 14:41:20.443791 8868 raft_consensus.cc:3037] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:20.443754 9141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994a211e2ff64fed86b85b643538045e" candidate_uuid: "14f5ba50d8e54f0490dc670239d05163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bc95a0d3cd4b339026c59818e3fadf"
I20260928 14:41:20.443914 9141 raft_consensus.cc:3037] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:20.444733 8868 raft_consensus.cc:2454] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14f5ba50d8e54f0490dc670239d05163 in term 1.
I20260928 14:41:20.444821 9141 raft_consensus.cc:2454] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14f5ba50d8e54f0490dc670239d05163 in term 1.
I20260928 14:41:20.445007 8939 leader_election.cc:302] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f5ba50d8e54f0490dc670239d05163, bd640aa359054a81aa2e5011e31b16ec; no voters:
I20260928 14:41:20.445165 9286 raft_consensus.cc:2781] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:20.445407 9286 raft_consensus.cc:686] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 1 LEADER]: Becoming Leader. State: Replica: 14f5ba50d8e54f0490dc670239d05163, State: Running, Role: LEADER
I20260928 14:41:20.445713 9286 consensus_queue.cc:237] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } } peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } }
I20260928 14:41:20.446007 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.445800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.2:59295)
0928 14:41:20.445902 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.445953 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:41:20.447911 8738 catalog_manager.cc:5495] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14f5ba50d8e54f0490dc670239d05163 (127.1.201.2). New cstate: current_term: 1 leader_uuid: "14f5ba50d8e54f0490dc670239d05163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f5ba50d8e54f0490dc670239d05163" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 } health_report { overall_health: UNKNOWN } } }
I20260928 14:41:20.544771 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.541770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.1.201.2:39669)
0928 14:41:20.541907 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:20.541910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:20.541922 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:20.542012 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:20.542014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:20.542065 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:20.542157 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:20.542377 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.542841 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.543627 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:20.543836 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:20.543908 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:20.544115 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:20.544516 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:20.544593 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:20.544644 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:41:20.544684 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:41:20.547209 9141 raft_consensus.cc:1261] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf [term 1 FOLLOWER]: Refusing update from remote peer 14f5ba50d8e54f0490dc670239d05163: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:20.547407 8868 raft_consensus.cc:1261] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec [term 1 FOLLOWER]: Refusing update from remote peer 14f5ba50d8e54f0490dc670239d05163: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:20.547691 9295 consensus_queue.cc:1035] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bc95a0d3cd4b339026c59818e3fadf" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 37881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:20.547919 9286 consensus_queue.cc:1035] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd640aa359054a81aa2e5011e31b16ec" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:20.549254 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.548996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.3:36579)
0928 14:41:20.549116 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.549177 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 14:41:20.550549 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.550351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.1:60407)
0928 14:41:20.550432 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.550485 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 14:41:20.552578 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.552342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.2:50111)
0928 14:41:20.552461 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.552504 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 14:41:20.553295 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.553042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.3:35597)
0928 14:41:20.553183 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.553228 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 14:41:20.554085 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.553847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.2:59081)
0928 14:41:20.553982 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.554031 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 14:41:20.554471 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.554231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35941 (local address 127.1.201.1:36799)
0928 14:41:20.554356 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.554404 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 14:41:20.556998 9298 mvcc.cc:205] Tried to move back new op lower bound from 7334324144316825600 to 7334324143908425728. Current Snapshot: MvccSnapshot[applied={T|T < 7334324144316825600 or (T in {7334324144316825600})}]
I20260928 14:41:20.557169 9301 mvcc.cc:205] Tried to move back new op lower bound from 7334324144316825600 to 7334324143908425728. Current Snapshot: MvccSnapshot[applied={T|T < 7334324144316825600 or (T in {7334324144316825600})}]
I20260928 14:41:20.560590 8737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:20.560712 8737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:20.564265 8737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999ad913e9c04ad581d6db34d5705d82: Sending DeleteTablet for 3 replicas of tablet 994a211e2ff64fed86b85b643538045e
I20260928 14:41:20.564851 8848 tablet_service.cc:1514] Processing DeleteTablet for tablet 994a211e2ff64fed86b85b643538045e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:20 UTC) from {username='oryx', principal='oryx/127.1.201.60@KRBTEST.COM'} at 127.0.0.1:43190
I20260928 14:41:20.564936 8985 tablet_service.cc:1514] Processing DeleteTablet for tablet 994a211e2ff64fed86b85b643538045e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:20 UTC) from {username='oryx', principal='oryx/127.1.201.60@KRBTEST.COM'} at 127.0.0.1:38118
I20260928 14:41:20.565225 9308 tablet_replica.cc:330] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163: stopping tablet replica
I20260928 14:41:20.565267 9121 tablet_service.cc:1514] Processing DeleteTablet for tablet 994a211e2ff64fed86b85b643538045e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:20 UTC) from {username='oryx', principal='oryx/127.1.201.60@KRBTEST.COM'} at 127.0.0.1:41982
I20260928 14:41:20.565375 9308 raft_consensus.cc:2229] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:41:20.565590 9307 tablet_replica.cc:330] T 994a211e2ff64fed86b85b643538045e P bd640aa359054a81aa2e5011e31b16ec: stopping tablet replica
I20260928 14:41:20.565619 9308 raft_consensus.cc:2258] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:20.565802 9309 tablet_replica.cc:330] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf: stopping tablet replica
I20260928 14:41:20.565828 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 8697
I20260928 14:41:20.565915 9309 raft_consensus.cc:2229] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:20.567008 9308 ts_tablet_manager.cc:1905] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:20.567039 9309 raft_consensus.cc:2258] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:20.568629 9309 ts_tablet_manager.cc:1905] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:20.569525 9308 ts_tablet_manager.cc:1918] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:20.569648 9308 log.cc:1196] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/994a211e2ff64fed86b85b643538045e
I20260928 14:41:20.570077 9308 ts_tablet_manager.cc:1939] T 994a211e2ff64fed86b85b643538045e P 14f5ba50d8e54f0490dc670239d05163: Deleting consensus metadata
I20260928 14:41:20.570721 8720 catalog_manager.cc:4841] TS 14f5ba50d8e54f0490dc670239d05163 (127.1.201.2:39669): tablet 994a211e2ff64fed86b85b643538045e (table test-table [id=2696a4c7fd7c483dad279b19e44c6e9b]) successfully deleted
I20260928 14:41:20.571142 9309 ts_tablet_manager.cc:1918] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:20.571228 9309 log.cc:1196] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/994a211e2ff64fed86b85b643538045e
I20260928 14:41:20.571485 9309 ts_tablet_manager.cc:1939] T 994a211e2ff64fed86b85b643538045e P e1bc95a0d3cd4b339026c59818e3fadf: Deleting consensus metadata
I20260928 14:41:20.571998 8720 catalog_manager.cc:4841] TS e1bc95a0d3cd4b339026c59818e3fadf (127.1.201.3:37881): tablet 994a211e2ff64fed86b85b643538045e (table test-table [id=2696a4c7fd7c483dad279b19e44c6e9b]) successfully deleted
I20260928 14:41:20.577062 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 8922
I20260928 14:41:20.577836 9280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:20.577188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:33547 (local address 127.0.0.1:43192)
0928 14:41:20.577651 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:20.577728 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:33547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360}
W20260928 14:41:20.580276 8720 catalog_manager.cc:4584] TS bd640aa359054a81aa2e5011e31b16ec (127.1.201.1:33547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 994a211e2ff64fed86b85b643538045e: Network error: Client connection negotiation failed: client connection to 127.1.201.1:33547: connect: Connection refused (error 111)
I20260928 14:41:20.584010 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 9058
I20260928 14:41:20.589267 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 8416
I20260928 14:41:20.594828 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 8525
2026-09-28T14:41:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5522 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:41:20 dist-test-slave-0n27 krb5kdc[9312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:41:20 dist-test-slave-0n27 krb5kdc[9312](info): set up 2 sockets
Sep 28 14:41:20 dist-test-slave-0n27 krb5kdc[9312](info): commencing operation
krb5kdc: starting...
W20260928 14:41:21.586836 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:41:21 dist-test-slave-0n27 krb5kdc[9312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.62.keytab.
Entry for principal oryx/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.62.keytab.
I20260928 14:41:21.696997 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:38423
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.62.keytab
--principal=oryx/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:38423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:21.798182 9325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:21.798409 9325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:21.798498 9325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:21.802193 9325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:21.802284 9325 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:41:21.802318 9325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:21.802359 9325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:21.802400 9325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:21.802438 9325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:21.806902 9325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:38423
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.62.keytab
--principal=oryx/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:38423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:21.807701 9325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:21.808563 9325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:21.815961 9330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:21.816105 9325 server_base.cc:1034] running on GCE node
W20260928 14:41:21.816013 9331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:21.815974 9333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:21.816804 9325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:21.817811 9325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:21.819003 9325 hybrid_clock.cc:648] HybridClock initialized: now 1790606481818986 us; error 34 us; skew 500 ppm
Sep 28 14:41:21 dist-test-slave-0n27 krb5kdc[9312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606481, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:21.821416 9325 init.cc:377] Logged in from keytab as oryx/127.1.201.62@KRBTEST.COM (short username oryx)
I20260928 14:41:21.822234 9325 webserver.cc:460] Webserver started at http://127.1.201.62:37213/ using document root <none> and password file <none>
I20260928 14:41:21.822732 9325 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:21.822844 9325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:21.823057 9325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:21.824842 9325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "8b14ab93a1e54af7a6fb186a8ef130bf"
format_stamp: "Formatted at 2026-09-28 14:41:21 on dist-test-slave-0n27"
server_key: "14c083de0a7a9e62db95c7b14bc1d723"
server_key_iv: "32d0fe73efc2e60e4c86b3c68682cab7"
server_key_version: "encryptionkey@0"
I20260928 14:41:21.825343 9325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "8b14ab93a1e54af7a6fb186a8ef130bf"
format_stamp: "Formatted at 2026-09-28 14:41:21 on dist-test-slave-0n27"
server_key: "14c083de0a7a9e62db95c7b14bc1d723"
server_key_iv: "32d0fe73efc2e60e4c86b3c68682cab7"
server_key_version: "encryptionkey@0"
I20260928 14:41:21.828775 9325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 14:41:21.830986 9340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:21.832162 9325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:21.832302 9325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "8b14ab93a1e54af7a6fb186a8ef130bf"
format_stamp: "Formatted at 2026-09-28 14:41:21 on dist-test-slave-0n27"
server_key: "14c083de0a7a9e62db95c7b14bc1d723"
server_key_iv: "32d0fe73efc2e60e4c86b3c68682cab7"
server_key_version: "encryptionkey@0"
I20260928 14:41:21.832401 9325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:21.843132 9325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:21.845502 9325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:21.845705 9325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:21.852437 9325 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:38423
I20260928 14:41:21.852447 9402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:38423 every 8 connection(s)
I20260928 14:41:21.853212 9325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:21.855513 9403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:21.860121 9403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf: Bootstrap starting.
I20260928 14:41:21.862012 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 9325
I20260928 14:41:21.862140 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:21.862202 9403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:21.862425 1828 external_mini_cluster.cc:1428] Setting key 3eeaa9f42050b448f1bfed9b61ebfd09
I20260928 14:41:21.862927 9403 log.cc:824] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:21.864866 9403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf: No bootstrap required, opened a new log
I20260928 14:41:21.869118 9403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b14ab93a1e54af7a6fb186a8ef130bf" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38423 } }
I20260928 14:41:21.869388 9403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:21.869451 9403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b14ab93a1e54af7a6fb186a8ef130bf, State: Initialized, Role: FOLLOWER
I20260928 14:41:21.869956 9403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b14ab93a1e54af7a6fb186a8ef130bf" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38423 } }
I20260928 14:41:21.870124 9403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:21.870206 9403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:21.870301 9403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:21.871069 9403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b14ab93a1e54af7a6fb186a8ef130bf" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38423 } }
I20260928 14:41:21.871399 9403 leader_election.cc:302] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b14ab93a1e54af7a6fb186a8ef130bf; no voters:
Sep 28 14:41:21 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:21.871687 9403 leader_election.cc:288] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:21.871783 9408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:21.871994 9408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [term 1 LEADER]: Becoming Leader. State: Replica: 8b14ab93a1e54af7a6fb186a8ef130bf, State: Running, Role: LEADER
I20260928 14:41:21.872340 9408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b14ab93a1e54af7a6fb186a8ef130bf" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38423 } }
I20260928 14:41:21.872954 9403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:21.873412 9408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b14ab93a1e54af7a6fb186a8ef130bf. Latest consensus state: current_term: 1 leader_uuid: "8b14ab93a1e54af7a6fb186a8ef130bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b14ab93a1e54af7a6fb186a8ef130bf" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38423 } } }
I20260928 14:41:21.873610 9408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:21.874054 9414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:21.875488 9409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b14ab93a1e54af7a6fb186a8ef130bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b14ab93a1e54af7a6fb186a8ef130bf" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38423 } } }
I20260928 14:41:21.875617 9409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:21.877630 9414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:21.877683 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:21.863359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.1.201.62:38423)
0928 14:41:21.863779 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:21.863796 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:21.863832 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:41:21.864551 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:21.864571 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:21.864915 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:21.865227 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:21.865844 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:21.867291 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:21.868632 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:21.869326 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:21.869489 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:21.872301 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:21.872307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:21.872320 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:21.872347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:21.874477 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:21.875033 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:21.875042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:21.875047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:21.875185 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:21.875535 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:21.875538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:21.875539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:21.876072 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:21.876294 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:21.876574 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 14:41:21.876858 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":109,"threads_started":1}
I20260928 14:41:21.883052 9414 catalog_manager.cc:1347] Generated new cluster ID: 2a34f6c48cdc45c49c06d4d4949c133e
I20260928 14:41:21.883121 9414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:21.903221 9414 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:21.903999 9414 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:21.913074 9414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf: Generated new TSK 0
I20260928 14:41:21.913702 9414 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab.
Entry for principal oryx/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab.
I20260928 14:41:21.948171 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38423
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab
--principal=oryx/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:22.049427 9429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:22.049669 9429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:22.049813 9429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:22.053468 9429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:22.053562 9429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:22.053627 9429 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:41:22.053714 9429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:22.058241 9429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab
--principal=oryx/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.201.62:38423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:22.059099 9429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:22.059998 9429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:22.068704 9435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:22.068737 9434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:22.068845 9437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:22.068737 9429 server_base.cc:1034] running on GCE node
I20260928 14:41:22.069244 9429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:22.069732 9429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:22.070878 9429 hybrid_clock.cc:648] HybridClock initialized: now 1790606482070866 us; error 22 us; skew 500 ppm
Sep 28 14:41:22 dist-test-slave-0n27 krb5kdc[9312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:22.073508 9429 init.cc:377] Logged in from keytab as oryx/127.1.201.1@KRBTEST.COM (short username oryx)
I20260928 14:41:22.074364 9429 webserver.cc:460] Webserver started at http://127.1.201.1:43705/ using document root <none> and password file <none>
I20260928 14:41:22.074823 9429 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:22.074901 9429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:22.075099 9429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:22.076754 9429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "9891ab96fbe1479492a10687489a04fa"
format_stamp: "Formatted at 2026-09-28 14:41:22 on dist-test-slave-0n27"
server_key: "676affe2d77d2db0742cb512b81a3bd1"
server_key_iv: "2be3a864ad41dc9f0162eaed45dbf61c"
server_key_version: "encryptionkey@0"
I20260928 14:41:22.077247 9429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "9891ab96fbe1479492a10687489a04fa"
format_stamp: "Formatted at 2026-09-28 14:41:22 on dist-test-slave-0n27"
server_key: "676affe2d77d2db0742cb512b81a3bd1"
server_key_iv: "2be3a864ad41dc9f0162eaed45dbf61c"
server_key_version: "encryptionkey@0"
I20260928 14:41:22.080487 9429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:22.082659 9444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:22.083701 9429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:22.083822 9429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "9891ab96fbe1479492a10687489a04fa"
format_stamp: "Formatted at 2026-09-28 14:41:22 on dist-test-slave-0n27"
server_key: "676affe2d77d2db0742cb512b81a3bd1"
server_key_iv: "2be3a864ad41dc9f0162eaed45dbf61c"
server_key_version: "encryptionkey@0"
I20260928 14:41:22.083920 9429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:22.118158 9429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:22.121140 9429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:22.121330 9429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:22.122743 9429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:22.122812 9429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:22.122875 9429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:22.122913 9429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:41:22 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:22.132965 9429 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:41257
I20260928 14:41:22.133044 9559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:41257 every 8 connection(s)
I20260928 14:41:22.133868 9429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:22.135259 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.123389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:40695 (local address 127.1.201.62:38423)
0928 14:41:22.123502 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.123510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.124361 (+ 851us) server_negotiation.cc:407] Connection header received
0928 14:41:22.125322 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.125325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.125383 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.125438 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:22.126717 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.127358 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.128374 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.128648 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.128758 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.131743 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:22.131750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:22.131752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:22.131781 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:22.133273 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.133761 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.133765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.133767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.133877 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.134283 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.134286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.134288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.134478 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:22.134588 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.135053 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.135148 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:41:22.135967 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.123825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38423 (local address 127.1.201.1:40695)
0928 14:41:22.124235 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.124271 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.125081 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.125868 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.125874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.126148 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:22.126574 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.126586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.127477 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.127481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.128257 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.128266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.128795 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.128799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.129016 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.129585 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:22.129624 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:22.131590 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:22.133406 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.133411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.133423 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.133660 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.134005 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.134008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.134009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.134171 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.134658 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:22.134662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:22.134863 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.134995 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.135208 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
I20260928 14:41:22.137156 9560 heartbeater.cc:347] Connected to a master server at 127.1.201.62:38423
I20260928 14:41:22.137296 9560 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:22.137712 9560 heartbeater.cc:511] Master 127.1.201.62:38423 requested a full tablet report, sending...
I20260928 14:41:22.139070 9356 ts_manager.cc:194] Registered new tserver with Master: 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257)
I20260928 14:41:22.140465 9356 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:40695
I20260928 14:41:22.143672 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 9429
I20260928 14:41:22.143808 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:22.144075 1828 external_mini_cluster.cc:1428] Setting key 4d40d5c8fd57079a5e069f38923011fb
I20260928 14:41:22.145206 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.139188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:33841 (local address 127.1.201.62:38423)
0928 14:41:22.139289 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.139293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.139457 (+ 164us) server_negotiation.cc:407] Connection header received
0928 14:41:22.139535 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.139538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.139575 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.139648 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:22.140013 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.140643 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.141437 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.141643 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.141742 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.142689 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:22.142692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:22.142694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:22.142715 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:22.143717 (+ 1002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.144129 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.144134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.144135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.144210 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.144483 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.144486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.144487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.144701 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:22.144774 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.145028 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.145111 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:41:22.145295 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.139102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38423 (local address 127.1.201.1:33841)
0928 14:41:22.139378 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.139389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.139462 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.139693 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.139695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.139838 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:22.139917 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.139921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.140764 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.140766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.141329 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.141338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.141746 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.141749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.141853 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.142290 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:22.142300 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:22.142557 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:22.143853 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.143858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.143860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.144038 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.144303 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.144306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.144308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.144393 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.144793 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:22.144796 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:22.144876 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.144998 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.145113 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.1.201.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab.
Entry for principal oryx/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab.
I20260928 14:41:22.168762 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38423
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab
--principal=oryx/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:22.269491 9569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:22.269765 9569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:22.269851 9569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:22.273481 9569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:22.273551 9569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:22.273635 9569 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:41:22.273718 9569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:22.278772 9569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab
--principal=oryx/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.201.62:38423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:22.279564 9569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:22.280427 9569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:22.288928 9574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:22.288944 9575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:22.289219 9569 server_base.cc:1034] running on GCE node
W20260928 14:41:22.289083 9577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:22.289705 9569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:22.290263 9569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:22.291410 9569 hybrid_clock.cc:648] HybridClock initialized: now 1790606482291385 us; error 37 us; skew 500 ppm
Sep 28 14:41:22 dist-test-slave-0n27 krb5kdc[9312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:22.293758 9569 init.cc:377] Logged in from keytab as oryx/127.1.201.2@KRBTEST.COM (short username oryx)
I20260928 14:41:22.294551 9569 webserver.cc:460] Webserver started at http://127.1.201.2:32951/ using document root <none> and password file <none>
I20260928 14:41:22.295015 9569 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:22.295064 9569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:22.295200 9569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:22.296953 9569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "6762c03b2039486fb9ecb50b070a7b64"
format_stamp: "Formatted at 2026-09-28 14:41:22 on dist-test-slave-0n27"
server_key: "9bad98d1e6cfa7ffef94000f77338f32"
server_key_iv: "ab789249a425bd428273b3a2f5a894c7"
server_key_version: "encryptionkey@0"
I20260928 14:41:22.297462 9569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "6762c03b2039486fb9ecb50b070a7b64"
format_stamp: "Formatted at 2026-09-28 14:41:22 on dist-test-slave-0n27"
server_key: "9bad98d1e6cfa7ffef94000f77338f32"
server_key_iv: "ab789249a425bd428273b3a2f5a894c7"
server_key_version: "encryptionkey@0"
I20260928 14:41:22.300808 9569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:22.302974 9584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:22.304072 9569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:22.304207 9569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "6762c03b2039486fb9ecb50b070a7b64"
format_stamp: "Formatted at 2026-09-28 14:41:22 on dist-test-slave-0n27"
server_key: "9bad98d1e6cfa7ffef94000f77338f32"
server_key_iv: "ab789249a425bd428273b3a2f5a894c7"
server_key_version: "encryptionkey@0"
I20260928 14:41:22.304311 9569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:22.323015 9569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:22.325287 9569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:22.325460 9569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:22.326834 9569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:22.326905 9569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:22.327001 9569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:22.327049 9569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:41:22 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:22.337183 9569 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:34059
I20260928 14:41:22.337460 9699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:34059 every 8 connection(s)
I20260928 14:41:22.338176 9569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:22.339939 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.327370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:50537 (local address 127.1.201.62:38423)
0928 14:41:22.327514 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.327518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.328322 (+ 804us) server_negotiation.cc:407] Connection header received
0928 14:41:22.329189 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.329196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.329258 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.329352 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:22.330636 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.331343 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.333119 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.333347 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.333458 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.336367 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:22.336370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:22.336373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:22.336404 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:22.338052 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.338593 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.338598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.338600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.338696 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.339054 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.339058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.339059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.339238 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:22.339317 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.339729 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.339818 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:41:22.340543 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.327754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38423 (local address 127.1.201.2:50537)
0928 14:41:22.328157 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.328208 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.328985 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.329621 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.329631 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.330022 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:22.330428 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.330440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.331475 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.331480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.332970 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.332983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.333501 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.333504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.333687 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.334184 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:22.334201 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:22.336191 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:22.338152 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.338156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.338206 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.338467 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.338809 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.338812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.338815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.338971 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.339329 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:22.339333 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:22.339515 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.339651 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.339884 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":111,"threads_started":1}
I20260928 14:41:22.341490 9700 heartbeater.cc:347] Connected to a master server at 127.1.201.62:38423
I20260928 14:41:22.341609 9700 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:22.342064 9700 heartbeater.cc:511] Master 127.1.201.62:38423 requested a full tablet report, sending...
I20260928 14:41:22.342988 9357 ts_manager.cc:194] Registered new tserver with Master: 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059)
I20260928 14:41:22.343655 9357 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:50537
I20260928 14:41:22.343988 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 9569
I20260928 14:41:22.344110 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:22.344422 1828 external_mini_cluster.cc:1428] Setting key b187b2fbcce58dd5c5be2a255d19a518
I20260928 14:41:22.350822 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.343369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38423 (local address 127.1.201.2:58773)
0928 14:41:22.343635 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.343649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.343744 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.344107 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.344110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.344305 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:22.344417 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.344424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.345304 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.345306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.346104 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.346117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.346563 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.346570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.346722 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.347136 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:22.347166 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:22.347592 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:22.349373 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.349375 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.349377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.349559 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.349871 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.349874 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.349875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.349949 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.350463 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:22.350465 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:22.350532 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.350610 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.350700 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
I20260928 14:41:22.350971 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.343487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:58773 (local address 127.1.201.62:38423)
0928 14:41:22.343589 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.343592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.343723 (+ 131us) server_negotiation.cc:407] Connection header received
0928 14:41:22.343845 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.343851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.343896 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.343992 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:22.344563 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.345185 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.346223 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.346440 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.346530 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.347773 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:22.347777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:22.347779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:22.347807 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:22.349267 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.349654 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.349660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.349665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.349753 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.350062 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.350068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.350073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.350361 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:22.350470 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.350691 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.350802 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for oryx/127.1.201.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab.
Entry for principal oryx/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab.
I20260928 14:41:22.369756 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:38423
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab
--principal=oryx/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:22.471371 9709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:22.471601 9709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:22.471685 9709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:22.475564 9709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:22.475641 9709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:22.475713 9709 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:41:22.475785 9709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:22.480485 9709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab
--principal=oryx/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.201.62:38423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:22.481326 9709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:22.482183 9709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:22.490725 9715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:22.490749 9717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:22.490906 9709 server_base.cc:1034] running on GCE node
W20260928 14:41:22.490717 9714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:22.491351 9709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:22.491889 9709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:22.493037 9709 hybrid_clock.cc:648] HybridClock initialized: now 1790606482492988 us; error 61 us; skew 500 ppm
Sep 28 14:41:22 dist-test-slave-0n27 krb5kdc[9312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:22.495447 9709 init.cc:377] Logged in from keytab as oryx/127.1.201.3@KRBTEST.COM (short username oryx)
I20260928 14:41:22.496235 9709 webserver.cc:460] Webserver started at http://127.1.201.3:44783/ using document root <none> and password file <none>
I20260928 14:41:22.496722 9709 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:22.496807 9709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:22.497015 9709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:22.498921 9709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "7216319cfbe94d438e63a2b14976f23a"
format_stamp: "Formatted at 2026-09-28 14:41:22 on dist-test-slave-0n27"
server_key: "e6e69a3acc0c676c79fdc3d4071f2347"
server_key_iv: "bc0f39be8aba6fd63b77345872f30faa"
server_key_version: "encryptionkey@0"
I20260928 14:41:22.499449 9709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "7216319cfbe94d438e63a2b14976f23a"
format_stamp: "Formatted at 2026-09-28 14:41:22 on dist-test-slave-0n27"
server_key: "e6e69a3acc0c676c79fdc3d4071f2347"
server_key_iv: "bc0f39be8aba6fd63b77345872f30faa"
server_key_version: "encryptionkey@0"
I20260928 14:41:22.502683 9709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:41:22.504771 9724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:22.505899 9709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:22.506002 9709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "7216319cfbe94d438e63a2b14976f23a"
format_stamp: "Formatted at 2026-09-28 14:41:22 on dist-test-slave-0n27"
server_key: "e6e69a3acc0c676c79fdc3d4071f2347"
server_key_iv: "bc0f39be8aba6fd63b77345872f30faa"
server_key_version: "encryptionkey@0"
I20260928 14:41:22.506093 9709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:22.539047 9709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:22.541519 9709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:22.541730 9709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:22.543006 9709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:22.543051 9709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:22.543084 9709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:22.543134 9709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:41:22 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:22.554296 9709 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:43607
I20260928 14:41:22.554368 9839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:43607 every 8 connection(s)
I20260928 14:41:22.555299 9709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:41:22.556784 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.543849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:54933 (local address 127.1.201.62:38423)
0928 14:41:22.543964 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.543975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.544929 (+ 954us) server_negotiation.cc:407] Connection header received
0928 14:41:22.545970 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.545974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.546040 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.546105 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:22.547715 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.548243 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.550013 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.550237 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.550342 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.553096 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:22.553099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:22.553103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:22.553130 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:22.554742 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.555259 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.555265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.555269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.555375 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.555813 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.555818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.555820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.556022 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:22.556124 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.556568 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.556660 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:41:22.557432 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.544340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38423 (local address 127.1.201.3:54933)
0928 14:41:22.544790 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.544844 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.545714 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.546517 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.546525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.546900 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:22.547555 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.547574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.548356 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.548359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.549863 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.549872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.550389 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.550392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.550598 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.551169 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:22.551186 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:22.552676 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:22.554850 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.554854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.554863 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.555157 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.555488 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.555491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.555493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.555676 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.556124 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:22.556130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:22.556370 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.556511 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.556722 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260928 14:41:22.558780 9840 heartbeater.cc:347] Connected to a master server at 127.1.201.62:38423
I20260928 14:41:22.558917 9840 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:22.559423 9840 heartbeater.cc:511] Master 127.1.201.62:38423 requested a full tablet report, sending...
I20260928 14:41:22.560485 9357 ts_manager.cc:194] Registered new tserver with Master: 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607)
I20260928 14:41:22.561057 9357 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:54933
I20260928 14:41:22.565239 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 9709
I20260928 14:41:22.565369 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:22.565657 1828 external_mini_cluster.cc:1428] Setting key ccccb010e6264d4653d7e9fe2d35096d
I20260928 14:41:22.567534 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.561076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:36537 (local address 127.1.201.62:38423)
0928 14:41:22.561181 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.561184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.561431 (+ 247us) server_negotiation.cc:407] Connection header received
0928 14:41:22.561518 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.561521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.561561 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.561672 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:22.562144 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.562591 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.563410 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.563587 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.563667 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.564542 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:22.564545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:22.564547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:22.564564 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:22.565885 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.566332 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.566335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.566336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.566415 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.566725 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.566728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.566729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.566953 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:22.567028 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.567297 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.567375 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:41:22.567549 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.561039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38423 (local address 127.1.201.3:36537)
0928 14:41:22.561345 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.561360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.561446 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.561697 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.561699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.561923 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:22.562032 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.562040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.562732 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.562736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.563318 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.563327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.563684 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.563686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.563744 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.564162 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:22.564181 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:22.564431 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:22.565987 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.565990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.565993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.566229 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.566537 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:22.566540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:22.566541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:22.566617 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:22.567046 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:22.567049 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:22.567142 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.567251 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.567372 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260928 14:41:22.567708 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:41:22 dist-test-slave-0n27 krb5kdc[9312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:41:22 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:22 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:41:22.590747 9407 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:22.584716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.1.201.62:38423)
0928 14:41:22.584847 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.584851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.584861 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:41:22.584892 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.584894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.584927 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.585011 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:22.585664 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.586207 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.587101 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.587332 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.587433 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.590619 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45582: BlockingRecv error: recv got EOF from 127.0.0.1:45582 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 14:41:22 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:22.599311 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.592062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.1.201.62:38423)
0928 14:41:22.592192 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.592196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.592300 (+ 104us) server_negotiation.cc:407] Connection header received
0928 14:41:22.592435 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.592438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.592480 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.592558 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:22.593126 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.593581 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.594393 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.594594 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.594697 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.596798 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:22.596801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:22.596803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:22.596823 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:22.598024 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.598392 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.598394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.598395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.598459 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:22.598710 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:22.598713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:22.598714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:22.598856 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:22.598947 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.599124 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.599215 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:41:22.601711 9357 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:41:22.604532 9357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:41:22.617764 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.612671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.1.201.2:34059)
0928 14:41:22.613594 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.613611 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.613642 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:41:22.613708 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.613714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.613893 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.614044 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:22.614273 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.615091 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.616501 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.617142 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.617261 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.617421 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.617548 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.617626 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":96,"threads_started":1}
I20260928 14:41:22.617754 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.612128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:34059 (local address 127.0.0.1:37078)
0928 14:41:22.613076 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.613092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.613179 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.614039 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.614042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.614058 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:22.614160 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.614166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.615252 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.615256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.616411 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.616421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.617262 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.617265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.617367 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.617417 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.617512 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.617625 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":8,"thread_start_us":85,"threads_started":1}
I20260928 14:41:22.618032 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.612395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:43607 (local address 127.0.0.1:55248)
0928 14:41:22.613226 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.613240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.613321 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.613832 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.613835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.613853 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:22.613928 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.613933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.615167 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.615171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.616254 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.616265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.617481 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.617484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.617671 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.617731 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.617819 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.617905 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":125,"thread_start_us":75,"threads_started":1}
I20260928 14:41:22.618144 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.612122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.1.201.1:41257)
0928 14:41:22.612735 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.612754 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.612780 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:41:22.612854 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.612859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.613042 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.613212 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:22.613668 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.614839 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.616436 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.617159 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.617260 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.617420 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.617905 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.617992 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":112,"threads_started":1}
I20260928 14:41:22.618428 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.611944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:41257 (local address 127.0.0.1:49520)
0928 14:41:22.612579 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.612623 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.612735 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.613391 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.613396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.613417 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:22.613540 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.613550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.614969 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.614972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.616333 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.616344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.617481 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.617484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.617669 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.617728 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.617826 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.618317 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":526,"thread_start_us":74,"threads_started":1}
I20260928 14:41:22.618448 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.612726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.1.201.3:43607)
0928 14:41:22.613085 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.613098 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.613403 (+ 305us) server_negotiation.cc:407] Connection header received
0928 14:41:22.613480 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.613488 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.613674 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.613836 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:22.614054 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.615064 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.616354 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.617156 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.617260 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.617420 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.618242 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.618330 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":92,"threads_started":1}
I20260928 14:41:22.620776 9774 tablet_service.cc:1467] Processing CreateTablet for tablet 679a48b1c495403799cb8b3828470393 (DEFAULT_TABLE table=test-table [id=05f60df31ad34d2b889178df724e0385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:22.620890 9494 tablet_service.cc:1467] Processing CreateTablet for tablet 679a48b1c495403799cb8b3828470393 (DEFAULT_TABLE table=test-table [id=05f60df31ad34d2b889178df724e0385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:22.620890 9634 tablet_service.cc:1467] Processing CreateTablet for tablet 679a48b1c495403799cb8b3828470393 (DEFAULT_TABLE table=test-table [id=05f60df31ad34d2b889178df724e0385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:22.621590 9774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679a48b1c495403799cb8b3828470393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:22.621806 9634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679a48b1c495403799cb8b3828470393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:22.621809 9494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679a48b1c495403799cb8b3828470393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:22.628144 9868 tablet_bootstrap.cc:492] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: Bootstrap starting.
I20260928 14:41:22.628916 9869 tablet_bootstrap.cc:492] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: Bootstrap starting.
I20260928 14:41:22.629316 9870 tablet_bootstrap.cc:492] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: Bootstrap starting.
I20260928 14:41:22.630424 9868 tablet_bootstrap.cc:654] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:22.631246 9868 log.cc:824] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:22.631348 9870 tablet_bootstrap.cc:654] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:22.631407 9869 tablet_bootstrap.cc:654] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:22.632087 9870 log.cc:824] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:22.632154 9869 log.cc:824] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:22.633215 9868 tablet_bootstrap.cc:492] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: No bootstrap required, opened a new log
I20260928 14:41:22.633388 9868 ts_tablet_manager.cc:1397] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:41:22.633872 9870 tablet_bootstrap.cc:492] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: No bootstrap required, opened a new log
I20260928 14:41:22.634033 9870 ts_tablet_manager.cc:1397] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:41:22.634754 9869 tablet_bootstrap.cc:492] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: No bootstrap required, opened a new log
I20260928 14:41:22.634871 9869 ts_tablet_manager.cc:1397] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 14:41:22.637339 9868 raft_consensus.cc:348] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.637569 9868 raft_consensus.cc:374] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:22.637612 9868 raft_consensus.cc:729] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7216319cfbe94d438e63a2b14976f23a, State: Initialized, Role: FOLLOWER
I20260928 14:41:22.637655 9870 raft_consensus.cc:348] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.638005 9870 raft_consensus.cc:374] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:22.638085 9870 raft_consensus.cc:729] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9891ab96fbe1479492a10687489a04fa, State: Initialized, Role: FOLLOWER
I20260928 14:41:22.638089 9868 consensus_queue.cc:260] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.637997 9869 raft_consensus.cc:348] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.638242 9869 raft_consensus.cc:374] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:22.638299 9869 raft_consensus.cc:729] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6762c03b2039486fb9ecb50b070a7b64, State: Initialized, Role: FOLLOWER
I20260928 14:41:22.638538 9870 consensus_queue.cc:260] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.638716 9869 consensus_queue.cc:260] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.639043 9868 ts_tablet_manager.cc:1428] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:41:22.639134 9560 heartbeater.cc:503] Master 127.1.201.62:38423 was elected leader, sending a full tablet report...
I20260928 14:41:22.639390 9870 ts_tablet_manager.cc:1428] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:41:22.639398 9840 heartbeater.cc:503] Master 127.1.201.62:38423 was elected leader, sending a full tablet report...
I20260928 14:41:22.639794 9869 ts_tablet_manager.cc:1428] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:41:22.640100 9700 heartbeater.cc:503] Master 127.1.201.62:38423 was elected leader, sending a full tablet report...
I20260928 14:41:22.797127 9876 raft_consensus.cc:482] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:41:22.797356 9876 raft_consensus.cc:504] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.798601 9876 leader_election.cc:288] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257), 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607)
I20260928 14:41:22.802955 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.798784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:41257 (local address 127.1.201.2:49953)
0928 14:41:22.798918 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.798935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.799021 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.799313 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.799319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.799343 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:22.799471 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.799479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.800463 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.800465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.801728 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.801738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.802537 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.802540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.802672 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.802694 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.802754 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.802806 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:41:22.803047 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.798956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:49953 (local address 127.1.201.1:41257)
0928 14:41:22.799082 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.799086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.799103 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:41:22.799163 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.799165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.799204 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.799268 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:22.799586 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.800364 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.801877 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.802415 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.802508 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.802582 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.802809 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.802890 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:41:22.803303 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.798988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:45399 (local address 127.1.201.3:43607)
0928 14:41:22.799117 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.799121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.799400 (+ 279us) server_negotiation.cc:407] Connection header received
0928 14:41:22.799471 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.799473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.799512 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.799605 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:22.799852 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.800528 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.802243 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.802776 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.802871 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.802921 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.803133 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.803219 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:41:22.803628 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.798868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:43607 (local address 127.1.201.2:45399)
0928 14:41:22.799291 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:22.799303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:22.799393 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:22.799627 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:22.799629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:22.799642 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:22.799749 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.799753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.800611 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.800614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.802143 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:22.802152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.802883 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.802886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:22.802979 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.803006 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:41:22.803450 (+ 444us) client_negotiation.cc:240] Negotiation successful
0928 14:41:22.803516 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":113,"threads_started":1}
I20260928 14:41:22.803609 9514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679a48b1c495403799cb8b3828470393" candidate_uuid: "6762c03b2039486fb9ecb50b070a7b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9891ab96fbe1479492a10687489a04fa" is_pre_election: true
I20260928 14:41:22.803915 9514 raft_consensus.cc:2454] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6762c03b2039486fb9ecb50b070a7b64 in term 0.
I20260928 14:41:22.804224 9794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679a48b1c495403799cb8b3828470393" candidate_uuid: "6762c03b2039486fb9ecb50b070a7b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7216319cfbe94d438e63a2b14976f23a" is_pre_election: true
I20260928 14:41:22.804383 9586 leader_election.cc:302] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6762c03b2039486fb9ecb50b070a7b64, 9891ab96fbe1479492a10687489a04fa; no voters:
I20260928 14:41:22.804497 9794 raft_consensus.cc:2454] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6762c03b2039486fb9ecb50b070a7b64 in term 0.
I20260928 14:41:22.804661 9876 raft_consensus.cc:2781] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:41:22.804744 9876 raft_consensus.cc:482] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:41:22.804800 9876 raft_consensus.cc:3037] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:22.805869 9876 raft_consensus.cc:504] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.806156 9876 leader_election.cc:288] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [CANDIDATE]: Term 1 election: Requested vote from peers 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257), 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607)
I20260928 14:41:22.806454 9794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679a48b1c495403799cb8b3828470393" candidate_uuid: "6762c03b2039486fb9ecb50b070a7b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7216319cfbe94d438e63a2b14976f23a"
I20260928 14:41:22.806465 9514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679a48b1c495403799cb8b3828470393" candidate_uuid: "6762c03b2039486fb9ecb50b070a7b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9891ab96fbe1479492a10687489a04fa"
I20260928 14:41:22.806567 9514 raft_consensus.cc:3037] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Advancing to term 1
W20260928 14:41:22.806564 9841 tablet.cc:2369] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:41:22.806567 9794 raft_consensus.cc:3037] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:22.807734 9514 raft_consensus.cc:2454] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6762c03b2039486fb9ecb50b070a7b64 in term 1.
I20260928 14:41:22.808029 9586 leader_election.cc:302] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6762c03b2039486fb9ecb50b070a7b64, 9891ab96fbe1479492a10687489a04fa; no voters:
I20260928 14:41:22.808109 9794 raft_consensus.cc:2454] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6762c03b2039486fb9ecb50b070a7b64 in term 1.
I20260928 14:41:22.808190 9876 raft_consensus.cc:2781] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:22.808347 9876 raft_consensus.cc:686] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 LEADER]: Becoming Leader. State: Replica: 6762c03b2039486fb9ecb50b070a7b64, State: Running, Role: LEADER
I20260928 14:41:22.808701 9876 consensus_queue.cc:237] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.811712 9357 catalog_manager.cc:5495] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2). New cstate: current_term: 1 leader_uuid: "6762c03b2039486fb9ecb50b070a7b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } health_report { overall_health: UNKNOWN } } }
W20260928 14:41:22.839416 9701 tablet.cc:2369] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:41:22.848733 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:22.845388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.1.201.2:34059)
0928 14:41:22.845491 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:22.845497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:22.845528 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:41:22.845694 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:22.845697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:22.845752 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:22.845874 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:22.846090 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.846576 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.847396 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:22.847624 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:22.847729 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:22.847983 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:22.848367 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:22.848486 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:22.848567 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 14:41:22.848614 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:41:22.853899 9794 raft_consensus.cc:1261] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Refusing update from remote peer 6762c03b2039486fb9ecb50b070a7b64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:22.853901 9514 raft_consensus.cc:1261] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Refusing update from remote peer 6762c03b2039486fb9ecb50b070a7b64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:22.854488 9876 consensus_queue.cc:1035] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:22.854773 9878 consensus_queue.cc:1035] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:22.860260 9881 mvcc.cc:205] Tried to move back new op lower bound from 7334324153763553280 to 7334324153587355648. Current Snapshot: MvccSnapshot[applied={T|T < 7334324153763553280}]
I20260928 14:41:22.862535 9882 mvcc.cc:205] Tried to move back new op lower bound from 7334324153763553280 to 7334324153587355648. Current Snapshot: MvccSnapshot[applied={T|T < 7334324153763553280}]
I20260928 14:41:22.863991 9883 mvcc.cc:205] Tried to move back new op lower bound from 7334324153763553280 to 7334324153587355648. Current Snapshot: MvccSnapshot[applied={T|T < 7334324153763553280}]
I20260928 14:41:22.871304 9357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:22.871414 9357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:22.873446 9357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf: Sending DeleteTablet for 3 replicas of tablet 679a48b1c495403799cb8b3828470393
I20260928 14:41:22.874147 9494 tablet_service.cc:1514] Processing DeleteTablet for tablet 679a48b1c495403799cb8b3828470393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:22 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:49520
I20260928 14:41:22.874164 9634 tablet_service.cc:1514] Processing DeleteTablet for tablet 679a48b1c495403799cb8b3828470393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:22 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:37078
I20260928 14:41:22.874276 9774 tablet_service.cc:1514] Processing DeleteTablet for tablet 679a48b1c495403799cb8b3828470393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:22 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:55248
I20260928 14:41:22.874527 9890 tablet_replica.cc:330] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: stopping tablet replica
I20260928 14:41:22.874663 9891 tablet_replica.cc:330] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: stopping tablet replica
I20260928 14:41:22.874800 9890 raft_consensus.cc:2229] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:22.874989 9891 raft_consensus.cc:2229] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:41:22.875064 9890 raft_consensus.cc:2258] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:22.875252 9357 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:41:22.875442 9891 raft_consensus.cc:2258] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:41:22.875615 9357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:41:22.875795 9892 tablet_replica.cc:330] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: stopping tablet replica
I20260928 14:41:22.876029 9892 raft_consensus.cc:2229] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:22.876297 9892 raft_consensus.cc:2258] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:22.877341 9891 ts_tablet_manager.cc:1905] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:22.877607 9890 ts_tablet_manager.cc:1905] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:22.879945 9891 ts_tablet_manager.cc:1918] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:22.880030 9891 log.cc:1196] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/679a48b1c495403799cb8b3828470393
I20260928 14:41:22.880117 9494 tablet_service.cc:1467] Processing CreateTablet for tablet bc9324893f2946979624c2d9c58fb73c (DEFAULT_TABLE table=test-table [id=a58611747c494ccb86bc3afab09c44b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:22.880316 9891 ts_tablet_manager.cc:1939] T 679a48b1c495403799cb8b3828470393 P 6762c03b2039486fb9ecb50b070a7b64: Deleting consensus metadata
I20260928 14:41:22.880358 9494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9324893f2946979624c2d9c58fb73c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:22.880367 9634 tablet_service.cc:1467] Processing CreateTablet for tablet bc9324893f2946979624c2d9c58fb73c (DEFAULT_TABLE table=test-table [id=a58611747c494ccb86bc3afab09c44b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:22.880564 9634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9324893f2946979624c2d9c58fb73c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:22.881263 9343 catalog_manager.cc:4841] TS 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059): tablet 679a48b1c495403799cb8b3828470393 (table test-table [id=05f60df31ad34d2b889178df724e0385]) successfully deleted
I20260928 14:41:22.881884 9774 tablet_service.cc:1467] Processing CreateTablet for tablet bc9324893f2946979624c2d9c58fb73c (DEFAULT_TABLE table=test-table [id=a58611747c494ccb86bc3afab09c44b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:22.882099 9774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9324893f2946979624c2d9c58fb73c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:22.882591 9870 tablet_bootstrap.cc:492] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: Bootstrap starting.
I20260928 14:41:22.883322 9870 tablet_bootstrap.cc:654] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:22.883812 9869 tablet_bootstrap.cc:492] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64: Bootstrap starting.
I20260928 14:41:22.884184 9868 tablet_bootstrap.cc:492] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a: Bootstrap starting.
I20260928 14:41:22.884738 9869 tablet_bootstrap.cc:654] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:22.885145 9868 tablet_bootstrap.cc:654] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:22.885810 9892 ts_tablet_manager.cc:1905] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:22.886073 9870 tablet_bootstrap.cc:492] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: No bootstrap required, opened a new log
I20260928 14:41:22.886157 9870 ts_tablet_manager.cc:1397] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 14:41:22.886924 9870 raft_consensus.cc:348] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.887030 9870 raft_consensus.cc:374] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:22.887051 9870 raft_consensus.cc:729] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9891ab96fbe1479492a10687489a04fa, State: Initialized, Role: FOLLOWER
I20260928 14:41:22.887169 9870 consensus_queue.cc:260] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.887429 9870 ts_tablet_manager.cc:1428] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:41:22.888432 9890 ts_tablet_manager.cc:1918] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:22.888545 9892 ts_tablet_manager.cc:1918] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:22.888635 9892 log.cc:1196] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/679a48b1c495403799cb8b3828470393
I20260928 14:41:22.888586 9890 log.cc:1196] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/679a48b1c495403799cb8b3828470393
I20260928 14:41:22.888926 9892 ts_tablet_manager.cc:1939] T 679a48b1c495403799cb8b3828470393 P 7216319cfbe94d438e63a2b14976f23a: Deleting consensus metadata
I20260928 14:41:22.889135 9890 ts_tablet_manager.cc:1939] T 679a48b1c495403799cb8b3828470393 P 9891ab96fbe1479492a10687489a04fa: Deleting consensus metadata
I20260928 14:41:22.890055 9344 catalog_manager.cc:4841] TS 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607): tablet 679a48b1c495403799cb8b3828470393 (table test-table [id=05f60df31ad34d2b889178df724e0385]) successfully deleted
I20260928 14:41:22.890307 9342 catalog_manager.cc:4841] TS 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257): tablet 679a48b1c495403799cb8b3828470393 (table test-table [id=05f60df31ad34d2b889178df724e0385]) successfully deleted
I20260928 14:41:22.891155 9869 tablet_bootstrap.cc:492] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64: No bootstrap required, opened a new log
I20260928 14:41:22.891247 9869 ts_tablet_manager.cc:1397] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 14:41:22.891417 9868 tablet_bootstrap.cc:492] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a: No bootstrap required, opened a new log
I20260928 14:41:22.891499 9868 ts_tablet_manager.cc:1397] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 14:41:22.891724 9869 raft_consensus.cc:348] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.891831 9869 raft_consensus.cc:374] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:22.891865 9869 raft_consensus.cc:729] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6762c03b2039486fb9ecb50b070a7b64, State: Initialized, Role: FOLLOWER
I20260928 14:41:22.891927 9868 raft_consensus.cc:348] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.892017 9868 raft_consensus.cc:374] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:22.891979 9869 consensus_queue.cc:260] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.892048 9868 raft_consensus.cc:729] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7216319cfbe94d438e63a2b14976f23a, State: Initialized, Role: FOLLOWER
I20260928 14:41:22.892154 9868 consensus_queue.cc:260] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:22.892248 9869 ts_tablet_manager.cc:1428] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:22.892403 9868 ts_tablet_manager.cc:1428] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260928 14:41:23.137148 9561 tablet.cc:2369] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:41:23.252267 9874 raft_consensus.cc:482] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:41:23.252403 9874 raft_consensus.cc:504] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.253505 9874 leader_election.cc:288] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257), 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059)
I20260928 14:41:23.258697 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.253886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:53805 (local address 127.1.201.1:41257)
0928 14:41:23.254013 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.254020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.254256 (+ 236us) server_negotiation.cc:407] Connection header received
0928 14:41:23.254393 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.254395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.254450 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.254524 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:23.254844 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.255546 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.257409 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.257971 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.258093 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.258150 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.258479 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.258542 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:41:23.258701 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.253934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:44905 (local address 127.1.201.2:34059)
0928 14:41:23.254087 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.254090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.254261 (+ 171us) server_negotiation.cc:407] Connection header received
0928 14:41:23.254392 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.254395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.254450 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.254524 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:23.254844 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.255546 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.257410 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.257971 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.258093 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.258154 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.258479 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.258544 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:41:23.258862 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.253780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:34059 (local address 127.1.201.3:44905)
0928 14:41:23.254182 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:23.254194 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:23.254287 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:23.254550 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:23.254553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:23.254576 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:23.254739 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.254747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.255713 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.255717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.257289 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.257298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.258120 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.258124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.258322 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.258360 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:41:23.258481 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 14:41:23.258655 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":314,"spinlock_wait_cycles":9344,"thread_start_us":80,"threads_started":1}
I20260928 14:41:23.258972 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.253771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:41257 (local address 127.1.201.3:53805)
0928 14:41:23.254146 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:23.254161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:23.254287 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:23.254550 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:23.254553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:23.254576 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:23.254747 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.254751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.255713 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.255717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.257286 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.257298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.258120 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.258124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.258318 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.258362 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:41:23.258481 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 14:41:23.258655 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":281,"spinlock_wait_cycles":27776,"thread_start_us":84,"threads_started":1}
I20260928 14:41:23.259403 9514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9324893f2946979624c2d9c58fb73c" candidate_uuid: "7216319cfbe94d438e63a2b14976f23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9891ab96fbe1479492a10687489a04fa" is_pre_election: true
I20260928 14:41:23.259451 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9324893f2946979624c2d9c58fb73c" candidate_uuid: "7216319cfbe94d438e63a2b14976f23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6762c03b2039486fb9ecb50b070a7b64" is_pre_election: true
I20260928 14:41:23.259578 9514 raft_consensus.cc:2454] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7216319cfbe94d438e63a2b14976f23a in term 0.
I20260928 14:41:23.259716 9654 raft_consensus.cc:2454] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7216319cfbe94d438e63a2b14976f23a in term 0.
I20260928 14:41:23.259943 9726 leader_election.cc:302] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7216319cfbe94d438e63a2b14976f23a, 9891ab96fbe1479492a10687489a04fa; no voters:
I20260928 14:41:23.260221 9874 raft_consensus.cc:2781] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:41:23.260290 9874 raft_consensus.cc:482] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:41:23.260341 9874 raft_consensus.cc:3037] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:23.261245 9874 raft_consensus.cc:504] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.261552 9874 leader_election.cc:288] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [CANDIDATE]: Term 1 election: Requested vote from peers 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257), 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059)
I20260928 14:41:23.262017 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9324893f2946979624c2d9c58fb73c" candidate_uuid: "7216319cfbe94d438e63a2b14976f23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6762c03b2039486fb9ecb50b070a7b64"
I20260928 14:41:23.262017 9514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9324893f2946979624c2d9c58fb73c" candidate_uuid: "7216319cfbe94d438e63a2b14976f23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9891ab96fbe1479492a10687489a04fa"
I20260928 14:41:23.262133 9654 raft_consensus.cc:3037] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:23.262135 9514 raft_consensus.cc:3037] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:23.263203 9514 raft_consensus.cc:2454] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7216319cfbe94d438e63a2b14976f23a in term 1.
I20260928 14:41:23.263324 9654 raft_consensus.cc:2454] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7216319cfbe94d438e63a2b14976f23a in term 1.
I20260928 14:41:23.263456 9726 leader_election.cc:302] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7216319cfbe94d438e63a2b14976f23a, 9891ab96fbe1479492a10687489a04fa; no voters:
I20260928 14:41:23.263600 9874 raft_consensus.cc:2781] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:23.263800 9874 raft_consensus.cc:686] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 1 LEADER]: Becoming Leader. State: Replica: 7216319cfbe94d438e63a2b14976f23a, State: Running, Role: LEADER
I20260928 14:41:23.264043 9874 consensus_queue.cc:237] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.265954 9357 catalog_manager.cc:5495] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7216319cfbe94d438e63a2b14976f23a (127.1.201.3). New cstate: current_term: 1 leader_uuid: "7216319cfbe94d438e63a2b14976f23a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } health_report { overall_health: HEALTHY } } }
Sep 28 14:41:23 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606481, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.62@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:23.366954 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.359282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:38423 (local address 127.0.0.1:45598)
0928 14:41:23.359558 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:23.359570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:23.359649 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:23.359928 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:23.359930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:23.360273 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:23.360363 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.360368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.361109 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.361111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.361654 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.361661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.362089 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.362091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.362149 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.362727 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:23.362739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:23.364103 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:23.365590 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:23.365602 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:23.365610 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:23.365917 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:23.366154 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:23.366156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:23.366157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:23.366231 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:23.366502 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:23.366507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:23.366593 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 14:41:23.366685 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:41:23.366801 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260928 14:41:23.366974 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.359372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.1.201.62:38423)
0928 14:41:23.359615 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.359622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.359635 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:41:23.359757 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.359759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.359809 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.359890 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:23.360485 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.361027 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.361747 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.362001 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.362074 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.364262 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:23.364266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:23.364269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:23.364296 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:23.365488 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:23.366000 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:23.366002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:23.366003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:23.366076 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:23.366284 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:23.366286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:23.366287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:23.366422 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:23.366488 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.366705 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.366849 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260928 14:41:23.370163 9357 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:45598:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:41:23.370828 9357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:41:23.375159 9494 tablet_service.cc:1467] Processing CreateTablet for tablet 2f128916eb0b48d28fad73cfa3c6e4cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66d61e865bad4489b37f87ded1b2238f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:41:23.375159 9634 tablet_service.cc:1467] Processing CreateTablet for tablet 2f128916eb0b48d28fad73cfa3c6e4cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66d61e865bad4489b37f87ded1b2238f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:41:23.375260 9774 tablet_service.cc:1467] Processing CreateTablet for tablet 2f128916eb0b48d28fad73cfa3c6e4cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66d61e865bad4489b37f87ded1b2238f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:41:23.375524 9494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f128916eb0b48d28fad73cfa3c6e4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:23.375524 9634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f128916eb0b48d28fad73cfa3c6e4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:23.375649 9774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f128916eb0b48d28fad73cfa3c6e4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:23.377908 9868 tablet_bootstrap.cc:492] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: Bootstrap starting.
I20260928 14:41:23.378235 9869 tablet_bootstrap.cc:492] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: Bootstrap starting.
I20260928 14:41:23.378235 9870 tablet_bootstrap.cc:492] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: Bootstrap starting.
I20260928 14:41:23.378810 9868 tablet_bootstrap.cc:654] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:23.379070 9869 tablet_bootstrap.cc:654] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:23.379094 9870 tablet_bootstrap.cc:654] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:23.380210 9869 tablet_bootstrap.cc:492] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: No bootstrap required, opened a new log
I20260928 14:41:23.380244 9868 tablet_bootstrap.cc:492] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: No bootstrap required, opened a new log
I20260928 14:41:23.380293 9869 ts_tablet_manager.cc:1397] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:23.380317 9868 ts_tablet_manager.cc:1397] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:23.380535 9870 tablet_bootstrap.cc:492] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: No bootstrap required, opened a new log
I20260928 14:41:23.380636 9870 ts_tablet_manager.cc:1397] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:41:23.380702 9868 raft_consensus.cc:348] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.380707 9869 raft_consensus.cc:348] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.380813 9869 raft_consensus.cc:374] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:23.380813 9868 raft_consensus.cc:374] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:23.380853 9868 raft_consensus.cc:729] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7216319cfbe94d438e63a2b14976f23a, State: Initialized, Role: FOLLOWER
I20260928 14:41:23.380853 9869 raft_consensus.cc:729] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6762c03b2039486fb9ecb50b070a7b64, State: Initialized, Role: FOLLOWER
I20260928 14:41:23.380971 9868 consensus_queue.cc:260] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.380971 9869 consensus_queue.cc:260] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.381083 9870 raft_consensus.cc:348] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.381202 9870 raft_consensus.cc:374] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:23.381218 9869 ts_tablet_manager.cc:1428] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:23.381227 9868 ts_tablet_manager.cc:1428] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:41:23.381237 9870 raft_consensus.cc:729] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9891ab96fbe1479492a10687489a04fa, State: Initialized, Role: FOLLOWER
I20260928 14:41:23.381187 9874 tablet_replica.cc:439] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } } }
I20260928 14:41:23.381316 9874 tablet_replica.cc:442] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:41:23.381347 9870 consensus_queue.cc:260] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.381376 9876 tablet_replica.cc:439] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } } }
I20260928 14:41:23.381486 9876 tablet_replica.cc:442] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:41:23.381580 9870 ts_tablet_manager.cc:1428] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260928 14:41:23.381862 9875 tablet_replica.cc:439] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } } }
I20260928 14:41:23.381989 9875 tablet_replica.cc:442] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:41:23.632550 9875 raft_consensus.cc:482] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:41:23.632691 9875 raft_consensus.cc:504] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.633728 9875 leader_election.cc:288] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059), 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607)
I20260928 14:41:23.637589 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.634070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:47499 (local address 127.1.201.2:34059)
0928 14:41:23.634215 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.634220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.634420 (+ 200us) server_negotiation.cc:407] Connection header received
0928 14:41:23.634569 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.634571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.634621 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.634703 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:23.634953 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.635572 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.636708 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.637086 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.637189 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.637264 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.637410 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.637475 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:41:23.637760 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.634029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:43607 (local address 127.1.201.1:36701)
0928 14:41:23.634429 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:23.634440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:23.634545 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:23.634799 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:23.634802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:23.634816 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:23.634924 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.634929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.635807 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.635810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.636654 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.636662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.637364 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.637367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.637470 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.637495 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:41:23.637570 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:41:23.637630 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":24,"thread_start_us":73,"threads_started":1}
I20260928 14:41:23.637802 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.634104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:36701 (local address 127.1.201.3:43607)
0928 14:41:23.634379 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.634384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.634505 (+ 121us) server_negotiation.cc:407] Connection header received
0928 14:41:23.634646 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.634649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.634700 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.634781 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:23.635018 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.635689 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.636762 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.637251 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.637365 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.637422 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.637598 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.637658 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":63,"threads_started":1}
I20260928 14:41:23.637821 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.634023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:34059 (local address 127.1.201.1:47499)
0928 14:41:23.634334 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:23.634345 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:23.634463 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:23.634713 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:23.634715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:23.634741 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:23.634852 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.634857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.635682 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.635687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.636593 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.636609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.637203 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.637207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.637289 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.637310 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:41:23.637396 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:41:23.637456 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260928 14:41:23.638177 9794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f128916eb0b48d28fad73cfa3c6e4cc" candidate_uuid: "9891ab96fbe1479492a10687489a04fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7216319cfbe94d438e63a2b14976f23a" is_pre_election: true
I20260928 14:41:23.638342 9794 raft_consensus.cc:2454] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9891ab96fbe1479492a10687489a04fa in term 0.
I20260928 14:41:23.638444 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f128916eb0b48d28fad73cfa3c6e4cc" candidate_uuid: "9891ab96fbe1479492a10687489a04fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6762c03b2039486fb9ecb50b070a7b64" is_pre_election: true
I20260928 14:41:23.638628 9654 raft_consensus.cc:2454] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9891ab96fbe1479492a10687489a04fa in term 0.
I20260928 14:41:23.638723 9448 leader_election.cc:302] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7216319cfbe94d438e63a2b14976f23a, 9891ab96fbe1479492a10687489a04fa; no voters:
I20260928 14:41:23.639016 9875 raft_consensus.cc:2781] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:41:23.639078 9875 raft_consensus.cc:482] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:41:23.639106 9875 raft_consensus.cc:3037] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:23.640221 9875 raft_consensus.cc:504] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.640592 9875 leader_election.cc:288] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [CANDIDATE]: Term 1 election: Requested vote from peers 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059), 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607)
I20260928 14:41:23.640820 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f128916eb0b48d28fad73cfa3c6e4cc" candidate_uuid: "9891ab96fbe1479492a10687489a04fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6762c03b2039486fb9ecb50b070a7b64"
I20260928 14:41:23.640925 9654 raft_consensus.cc:3037] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:23.640940 9794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f128916eb0b48d28fad73cfa3c6e4cc" candidate_uuid: "9891ab96fbe1479492a10687489a04fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7216319cfbe94d438e63a2b14976f23a"
I20260928 14:41:23.641055 9794 raft_consensus.cc:3037] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:23.641930 9654 raft_consensus.cc:2454] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9891ab96fbe1479492a10687489a04fa in term 1.
I20260928 14:41:23.642063 9794 raft_consensus.cc:2454] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9891ab96fbe1479492a10687489a04fa in term 1.
I20260928 14:41:23.642215 9447 leader_election.cc:302] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6762c03b2039486fb9ecb50b070a7b64, 9891ab96fbe1479492a10687489a04fa; no voters:
I20260928 14:41:23.642369 9875 raft_consensus.cc:2781] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:23.642570 9875 raft_consensus.cc:686] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [term 1 LEADER]: Becoming Leader. State: Replica: 9891ab96fbe1479492a10687489a04fa, State: Running, Role: LEADER
I20260928 14:41:23.642803 9875 consensus_queue.cc:237] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.643613 9913 tablet_replica.cc:439] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: TxnStatusTablet state changed. Reason: New leader 9891ab96fbe1479492a10687489a04fa. Latest consensus state: current_term: 1 leader_uuid: "9891ab96fbe1479492a10687489a04fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } } }
I20260928 14:41:23.643709 9913 tablet_replica.cc:442] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:41:23.644227 9915 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:41:23.644381 9915 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:41:23.644790 9357 catalog_manager.cc:5495] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9891ab96fbe1479492a10687489a04fa (127.1.201.1). New cstate: current_term: 1 leader_uuid: "9891ab96fbe1479492a10687489a04fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } health_report { overall_health: UNKNOWN } } }
I20260928 14:41:23.676482 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.672827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:41257 (local address 127.0.0.1:49524)
0928 14:41:23.673012 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:23.673032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:23.673110 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:23.673502 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:23.673504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:23.673529 (+ 25us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:41:23.673625 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.673629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.674363 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.674367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.675044 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.675052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.675447 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.675450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.675531 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.675579 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:41:23.676217 (+ 638us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:41:23.676231 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:41:23.676296 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:41:23.676374 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:41:23.676476 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.672915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.1.201.1:41257)
0928 14:41:23.673050 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.673054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.673068 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:41:23.673262 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.673265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.673388 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.673489 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:23.673728 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.674267 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.675153 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.675351 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.675446 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.675744 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:23.676114 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:23.676231 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.676305 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.676350 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:41:23.679024 9794 raft_consensus.cc:1261] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Refusing update from remote peer 9891ab96fbe1479492a10687489a04fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:23.679255 9654 raft_consensus.cc:1261] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Refusing update from remote peer 9891ab96fbe1479492a10687489a04fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:23.679458 9913 consensus_queue.cc:1035] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:23.679787 9875 consensus_queue.cc:1035] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:23.680878 9874 tablet_replica.cc:439] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: TxnStatusTablet state changed. Reason: New leader 9891ab96fbe1479492a10687489a04fa. Latest consensus state: current_term: 1 leader_uuid: "9891ab96fbe1479492a10687489a04fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } } }
I20260928 14:41:23.680975 9874 tablet_replica.cc:442] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:41:23.681255 9876 tablet_replica.cc:439] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: TxnStatusTablet state changed. Reason: New leader 9891ab96fbe1479492a10687489a04fa. Latest consensus state: current_term: 1 leader_uuid: "9891ab96fbe1479492a10687489a04fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } } }
I20260928 14:41:23.681360 9876 tablet_replica.cc:442] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:41:23.683158 9874 tablet_replica.cc:439] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9891ab96fbe1479492a10687489a04fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } } }
I20260928 14:41:23.683265 9874 tablet_replica.cc:442] T 2f128916eb0b48d28fad73cfa3c6e4cc P 7216319cfbe94d438e63a2b14976f23a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:41:23.684576 9875 tablet_replica.cc:439] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9891ab96fbe1479492a10687489a04fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } } }
I20260928 14:41:23.684662 9875 tablet_replica.cc:442] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:41:23.686071 9913 tablet_replica.cc:439] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9891ab96fbe1479492a10687489a04fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } } }
I20260928 14:41:23.686162 9913 tablet_replica.cc:442] T 2f128916eb0b48d28fad73cfa3c6e4cc P 9891ab96fbe1479492a10687489a04fa: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:41:23.686575 9876 tablet_replica.cc:439] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9891ab96fbe1479492a10687489a04fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } } }
I20260928 14:41:23.686669 9876 tablet_replica.cc:442] T 2f128916eb0b48d28fad73cfa3c6e4cc P 6762c03b2039486fb9ecb50b070a7b64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:41:23.692762 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.688534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.1.201.3:43607)
0928 14:41:23.688657 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.688662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.688892 (+ 230us) server_negotiation.cc:407] Connection header received
0928 14:41:23.689020 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.689023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.689086 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.689180 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:23.689973 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.690504 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.691498 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.691716 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.691821 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.692076 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:23.692434 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:23.692543 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.692604 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.692649 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:41:23.702519 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.699266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:42141 (local address 127.1.201.1:41257)
0928 14:41:23.699386 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.699390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.699546 (+ 156us) server_negotiation.cc:407] Connection header received
0928 14:41:23.699654 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.699656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.699698 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.699756 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:23.699981 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.700444 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.701558 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.701770 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.701924 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.702187 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:23.702254 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:23.702305 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.702405 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.702438 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:41:23.702598 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.699206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:41257 (local address 127.1.201.3:42141)
0928 14:41:23.699466 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:23.699479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:23.699574 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:23.699788 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:23.699791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:23.699798 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:41:23.699888 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.699893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.700537 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.700540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.701449 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.701459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.701937 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.701940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.702022 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.702068 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:41:23.702328 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:41:23.702334 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:41:23.702397 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:41:23.702461 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
I20260928 14:41:23.708990 9654 raft_consensus.cc:1261] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Refusing update from remote peer 7216319cfbe94d438e63a2b14976f23a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:23.709025 9514 raft_consensus.cc:1261] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Refusing update from remote peer 7216319cfbe94d438e63a2b14976f23a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:23.709474 9898 consensus_queue.cc:1035] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:23.709682 9874 consensus_queue.cc:1035] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:23.730104 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.726508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:52633 (local address 127.1.201.3:43607)
0928 14:41:23.726601 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.726605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.726779 (+ 174us) server_negotiation.cc:407] Connection header received
0928 14:41:23.726852 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.726855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.726896 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.726968 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:23.727159 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.727888 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.729038 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.729374 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.729468 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.729696 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:23.729770 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:23.729870 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.729951 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.729993 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:41:23.730285 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.726420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:43607 (local address 127.1.201.1:52633)
0928 14:41:23.726697 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:23.726709 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:23.726787 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:23.726980 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:23.726982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:23.726987 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:41:23.727069 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.727072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.728093 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.728096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.728925 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:23.728935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.729479 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.729481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:23.729545 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.729582 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:41:23.729890 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:41:23.729895 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:41:23.729939 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:41:23.729992 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260928 14:41:23.756325 9352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:23.756498 9352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:23.758524 9352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf: Sending DeleteTablet for 3 replicas of tablet bc9324893f2946979624c2d9c58fb73c
I20260928 14:41:23.759215 9634 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9324893f2946979624c2d9c58fb73c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:23 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:37078
I20260928 14:41:23.759249 9494 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9324893f2946979624c2d9c58fb73c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:23 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:49520
I20260928 14:41:23.759764 9774 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9324893f2946979624c2d9c58fb73c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:23 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:55248
I20260928 14:41:23.760002 9943 tablet_replica.cc:330] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a: stopping tablet replica
I20260928 14:41:23.760138 9943 raft_consensus.cc:2229] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:41:23.760375 9943 raft_consensus.cc:2258] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:23.760568 9941 tablet_replica.cc:330] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64: stopping tablet replica
I20260928 14:41:23.760668 9941 raft_consensus.cc:2229] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:23.760756 9941 raft_consensus.cc:2258] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:23.760944 9942 tablet_replica.cc:330] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: stopping tablet replica
I20260928 14:41:23.761054 9942 raft_consensus.cc:2229] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:23.761142 9942 raft_consensus.cc:2258] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:23.763922 9943 ts_tablet_manager.cc:1905] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:23.764039 9941 ts_tablet_manager.cc:1905] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:23.766592 9943 ts_tablet_manager.cc:1918] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:41:23.766716 9943 log.cc:1196] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/bc9324893f2946979624c2d9c58fb73c
I20260928 14:41:23.766803 9941 ts_tablet_manager.cc:1918] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:41:23.766883 9941 log.cc:1196] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/bc9324893f2946979624c2d9c58fb73c
I20260928 14:41:23.767104 9943 ts_tablet_manager.cc:1939] T bc9324893f2946979624c2d9c58fb73c P 7216319cfbe94d438e63a2b14976f23a: Deleting consensus metadata
I20260928 14:41:23.767138 9941 ts_tablet_manager.cc:1939] T bc9324893f2946979624c2d9c58fb73c P 6762c03b2039486fb9ecb50b070a7b64: Deleting consensus metadata
I20260928 14:41:23.767764 9344 catalog_manager.cc:4841] TS 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607): tablet bc9324893f2946979624c2d9c58fb73c (table test-table [id=a58611747c494ccb86bc3afab09c44b9]) successfully deleted
I20260928 14:41:23.767776 9343 catalog_manager.cc:4841] TS 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059): tablet bc9324893f2946979624c2d9c58fb73c (table test-table [id=a58611747c494ccb86bc3afab09c44b9]) successfully deleted
I20260928 14:41:23.769742 9942 ts_tablet_manager.cc:1905] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:23.772202 9942 ts_tablet_manager.cc:1918] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:41:23.772293 9942 log.cc:1196] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/bc9324893f2946979624c2d9c58fb73c
I20260928 14:41:23.772557 9942 ts_tablet_manager.cc:1939] T bc9324893f2946979624c2d9c58fb73c P 9891ab96fbe1479492a10687489a04fa: Deleting consensus metadata
I20260928 14:41:23.773144 9342 catalog_manager.cc:4841] TS 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257): tablet bc9324893f2946979624c2d9c58fb73c (table test-table [id=a58611747c494ccb86bc3afab09c44b9]) successfully deleted
Sep 28 14:41:23 dist-test-slave-0n27 krb5kdc[9312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:41:23 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:23 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:41:23.798852 9906 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:23.791461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.1.201.62:38423)
0928 14:41:23.791586 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.791590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.791835 (+ 245us) server_negotiation.cc:407] Connection header received
0928 14:41:23.791995 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.791998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.792055 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.792115 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:23.792899 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.793639 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.794888 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.795217 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.795314 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.798731 (+ 3417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45604: BlockingRecv error: recv got EOF from 127.0.0.1:45604 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 14:41:23 dist-test-slave-0n27 krb5kdc[9312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM
I20260928 14:41:23.810483 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.800562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.1.201.62:38423)
0928 14:41:23.800654 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.800658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.801026 (+ 368us) server_negotiation.cc:407] Connection header received
0928 14:41:23.801103 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.801106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.801150 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.801227 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:23.801870 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.802619 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.803710 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.804028 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.804126 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.806700 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:23.806703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:23.806706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:23.806729 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:23.808417 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:23.808893 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:23.808897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:23.808899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:23.809031 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:23.809386 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:23.809390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:23.809392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:23.809589 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:23.809686 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.810159 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.810329 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:41:23.813457 9352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:41:23.813910 9352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:41:23.818629 9634 tablet_service.cc:1467] Processing CreateTablet for tablet a2c4301310fe4402ae9c8ae79c37b3a9 (DEFAULT_TABLE table=test-table [id=0a00b9cb982a46eea75e1f0dbe949f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:23.818681 9774 tablet_service.cc:1467] Processing CreateTablet for tablet a2c4301310fe4402ae9c8ae79c37b3a9 (DEFAULT_TABLE table=test-table [id=0a00b9cb982a46eea75e1f0dbe949f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:23.818950 9774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c4301310fe4402ae9c8ae79c37b3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:23.818955 9634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c4301310fe4402ae9c8ae79c37b3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:23.820775 9869 tablet_bootstrap.cc:492] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64: Bootstrap starting.
I20260928 14:41:23.820968 9494 tablet_service.cc:1467] Processing CreateTablet for tablet a2c4301310fe4402ae9c8ae79c37b3a9 (DEFAULT_TABLE table=test-table [id=0a00b9cb982a46eea75e1f0dbe949f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:23.821158 9868 tablet_bootstrap.cc:492] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a: Bootstrap starting.
I20260928 14:41:23.821218 9494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c4301310fe4402ae9c8ae79c37b3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:23.821532 9869 tablet_bootstrap.cc:654] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:23.821969 9868 tablet_bootstrap.cc:654] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:23.823065 9868 tablet_bootstrap.cc:492] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a: No bootstrap required, opened a new log
I20260928 14:41:23.823164 9868 ts_tablet_manager.cc:1397] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:23.823686 9870 tablet_bootstrap.cc:492] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa: Bootstrap starting.
I20260928 14:41:23.823602 9868 raft_consensus.cc:348] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.823736 9868 raft_consensus.cc:374] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:23.823774 9868 raft_consensus.cc:729] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7216319cfbe94d438e63a2b14976f23a, State: Initialized, Role: FOLLOWER
I20260928 14:41:23.823886 9868 consensus_queue.cc:260] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.824151 9868 ts_tablet_manager.cc:1428] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:23.824415 9870 tablet_bootstrap.cc:654] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:23.824501 9869 tablet_bootstrap.cc:492] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64: No bootstrap required, opened a new log
I20260928 14:41:23.824574 9869 ts_tablet_manager.cc:1397] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 14:41:23.824921 9869 raft_consensus.cc:348] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.825033 9869 raft_consensus.cc:374] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:23.825067 9869 raft_consensus.cc:729] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6762c03b2039486fb9ecb50b070a7b64, State: Initialized, Role: FOLLOWER
I20260928 14:41:23.825192 9869 consensus_queue.cc:260] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.825475 9869 ts_tablet_manager.cc:1428] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:23.826648 9870 tablet_bootstrap.cc:492] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa: No bootstrap required, opened a new log
I20260928 14:41:23.826756 9870 ts_tablet_manager.cc:1397] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:41:23.827080 9870 raft_consensus.cc:348] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.827181 9870 raft_consensus.cc:374] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:23.827225 9870 raft_consensus.cc:729] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9891ab96fbe1479492a10687489a04fa, State: Initialized, Role: FOLLOWER
I20260928 14:41:23.827338 9870 consensus_queue.cc:260] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.827582 9870 ts_tablet_manager.cc:1428] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:41:23.893093 9937 raft_consensus.cc:482] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:41:23.893187 9937 raft_consensus.cc:504] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.893584 9937 leader_election.cc:288] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257), 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059)
I20260928 14:41:23.893947 9514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c4301310fe4402ae9c8ae79c37b3a9" candidate_uuid: "7216319cfbe94d438e63a2b14976f23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9891ab96fbe1479492a10687489a04fa" is_pre_election: true
I20260928 14:41:23.893951 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c4301310fe4402ae9c8ae79c37b3a9" candidate_uuid: "7216319cfbe94d438e63a2b14976f23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6762c03b2039486fb9ecb50b070a7b64" is_pre_election: true
I20260928 14:41:23.894088 9654 raft_consensus.cc:2454] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7216319cfbe94d438e63a2b14976f23a in term 0.
I20260928 14:41:23.894090 9514 raft_consensus.cc:2454] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7216319cfbe94d438e63a2b14976f23a in term 0.
I20260928 14:41:23.894337 9727 leader_election.cc:302] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6762c03b2039486fb9ecb50b070a7b64, 7216319cfbe94d438e63a2b14976f23a; no voters:
I20260928 14:41:23.894500 9937 raft_consensus.cc:2781] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:41:23.894555 9937 raft_consensus.cc:482] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:41:23.894613 9937 raft_consensus.cc:3037] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:23.895516 9937 raft_consensus.cc:504] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.895872 9937 leader_election.cc:288] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [CANDIDATE]: Term 1 election: Requested vote from peers 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257), 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059)
I20260928 14:41:23.896173 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c4301310fe4402ae9c8ae79c37b3a9" candidate_uuid: "7216319cfbe94d438e63a2b14976f23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6762c03b2039486fb9ecb50b070a7b64"
I20260928 14:41:23.896158 9514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c4301310fe4402ae9c8ae79c37b3a9" candidate_uuid: "7216319cfbe94d438e63a2b14976f23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9891ab96fbe1479492a10687489a04fa"
I20260928 14:41:23.896276 9514 raft_consensus.cc:3037] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:23.896276 9654 raft_consensus.cc:3037] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:23.897339 9654 raft_consensus.cc:2454] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7216319cfbe94d438e63a2b14976f23a in term 1.
I20260928 14:41:23.897339 9514 raft_consensus.cc:2454] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7216319cfbe94d438e63a2b14976f23a in term 1.
I20260928 14:41:23.897598 9727 leader_election.cc:302] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6762c03b2039486fb9ecb50b070a7b64, 7216319cfbe94d438e63a2b14976f23a; no voters:
I20260928 14:41:23.897814 9937 raft_consensus.cc:2781] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:23.897903 9937 raft_consensus.cc:686] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 1 LEADER]: Becoming Leader. State: Replica: 7216319cfbe94d438e63a2b14976f23a, State: Running, Role: LEADER
I20260928 14:41:23.898029 9937 consensus_queue.cc:237] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.899770 9352 catalog_manager.cc:5495] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7216319cfbe94d438e63a2b14976f23a (127.1.201.3). New cstate: current_term: 1 leader_uuid: "7216319cfbe94d438e63a2b14976f23a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } health_report { overall_health: HEALTHY } } }
I20260928 14:41:23.926294 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:23.923418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.1.201.3:43607)
0928 14:41:23.923548 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:23.923553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:23.923565 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:23.923716 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:23.923719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:23.923768 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:23.923843 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:23.924080 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.924531 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.925321 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:23.925529 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:23.925629 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:23.925882 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:23.925949 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:23.926022 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:23.926136 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 14:41:23.926178 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:41:23.927932 9514 raft_consensus.cc:1261] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Refusing update from remote peer 7216319cfbe94d438e63a2b14976f23a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:23.927949 9654 raft_consensus.cc:1261] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Refusing update from remote peer 7216319cfbe94d438e63a2b14976f23a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:23.928311 9874 consensus_queue.cc:1035] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:23.928431 9937 consensus_queue.cc:1035] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:23.934885 9352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:23.935020 9352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:23.937003 9352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf: Sending DeleteTablet for 3 replicas of tablet a2c4301310fe4402ae9c8ae79c37b3a9
I20260928 14:41:23.937522 9494 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c4301310fe4402ae9c8ae79c37b3a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:23 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:49520
I20260928 14:41:23.937675 9942 tablet_replica.cc:330] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa: stopping tablet replica
I20260928 14:41:23.937649 9634 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c4301310fe4402ae9c8ae79c37b3a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:23 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:37078
I20260928 14:41:23.937798 9941 tablet_replica.cc:330] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64: stopping tablet replica
I20260928 14:41:23.937794 9942 raft_consensus.cc:2229] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:23.937896 9941 raft_consensus.cc:2229] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:23.937923 9942 raft_consensus.cc:2258] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:23.937999 9941 raft_consensus.cc:2258] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:23.938067 9633 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c4301310fe4402ae9c8ae79c37b3a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:23 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:37078
I20260928 14:41:23.938112 9352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:41:23.938510 9352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:41:23.938596 9774 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c4301310fe4402ae9c8ae79c37b3a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:23 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:55248
I20260928 14:41:23.938580 9941 ts_tablet_manager.cc:1905] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:23.938766 9943 tablet_replica.cc:330] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a: stopping tablet replica
I20260928 14:41:23.938858 9943 raft_consensus.cc:2229] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:41:23.939016 9943 raft_consensus.cc:2258] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:23.939280 9942 ts_tablet_manager.cc:1905] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:23.940295 9943 ts_tablet_manager.cc:1905] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:23.941115 9941 ts_tablet_manager.cc:1918] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:23.941180 9941 log.cc:1196] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/a2c4301310fe4402ae9c8ae79c37b3a9
I20260928 14:41:23.941419 9941 ts_tablet_manager.cc:1939] T a2c4301310fe4402ae9c8ae79c37b3a9 P 6762c03b2039486fb9ecb50b070a7b64: Deleting consensus metadata
I20260928 14:41:23.941855 9942 ts_tablet_manager.cc:1918] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:23.941934 9942 log.cc:1196] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/a2c4301310fe4402ae9c8ae79c37b3a9
I20260928 14:41:23.942181 9942 ts_tablet_manager.cc:1939] T a2c4301310fe4402ae9c8ae79c37b3a9 P 9891ab96fbe1479492a10687489a04fa: Deleting consensus metadata
I20260928 14:41:23.942569 9943 ts_tablet_manager.cc:1918] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:23.942636 9943 log.cc:1196] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/a2c4301310fe4402ae9c8ae79c37b3a9
I20260928 14:41:23.942780 9342 catalog_manager.cc:4841] TS 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257): tablet a2c4301310fe4402ae9c8ae79c37b3a9 (table test-table [id=0a00b9cb982a46eea75e1f0dbe949f3e]) successfully deleted
I20260928 14:41:23.942857 9943 ts_tablet_manager.cc:1939] T a2c4301310fe4402ae9c8ae79c37b3a9 P 7216319cfbe94d438e63a2b14976f23a: Deleting consensus metadata
I20260928 14:41:23.943302 9344 catalog_manager.cc:4841] TS 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607): tablet a2c4301310fe4402ae9c8ae79c37b3a9 (table test-table [id=0a00b9cb982a46eea75e1f0dbe949f3e]) successfully deleted
I20260928 14:41:23.943358 9343 catalog_manager.cc:4841] TS 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059): tablet a2c4301310fe4402ae9c8ae79c37b3a9 (table test-table [id=0a00b9cb982a46eea75e1f0dbe949f3e]) successfully deleted
W20260928 14:41:23.943552 9343 catalog_manager.cc:4808] TS 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059): delete failed for tablet a2c4301310fe4402ae9c8ae79c37b3a9 because the tablet was not found. No further retry: Not found: Tablet not found: a2c4301310fe4402ae9c8ae79c37b3a9
I20260928 14:41:23.945029 9633 tablet_service.cc:1467] Processing CreateTablet for tablet 0d87a9fc438d4ebc83956df3ae601d27 (DEFAULT_TABLE table=test-table [id=312d62c89de4488eb9b230e6366ae45d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:23.945026 9494 tablet_service.cc:1467] Processing CreateTablet for tablet 0d87a9fc438d4ebc83956df3ae601d27 (DEFAULT_TABLE table=test-table [id=312d62c89de4488eb9b230e6366ae45d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:23.945086 9774 tablet_service.cc:1467] Processing CreateTablet for tablet 0d87a9fc438d4ebc83956df3ae601d27 (DEFAULT_TABLE table=test-table [id=312d62c89de4488eb9b230e6366ae45d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:23.945238 9633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d87a9fc438d4ebc83956df3ae601d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:23.945253 9494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d87a9fc438d4ebc83956df3ae601d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:23.945281 9774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d87a9fc438d4ebc83956df3ae601d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:23.947068 9869 tablet_bootstrap.cc:492] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64: Bootstrap starting.
I20260928 14:41:23.947182 9870 tablet_bootstrap.cc:492] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa: Bootstrap starting.
I20260928 14:41:23.947860 9869 tablet_bootstrap.cc:654] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:23.947860 9870 tablet_bootstrap.cc:654] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:23.948339 9868 tablet_bootstrap.cc:492] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a: Bootstrap starting.
I20260928 14:41:23.948951 9868 tablet_bootstrap.cc:654] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:23.948974 9870 tablet_bootstrap.cc:492] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa: No bootstrap required, opened a new log
I20260928 14:41:23.949069 9870 ts_tablet_manager.cc:1397] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:23.949236 9869 tablet_bootstrap.cc:492] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64: No bootstrap required, opened a new log
I20260928 14:41:23.949324 9869 ts_tablet_manager.cc:1397] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:23.949440 9870 raft_consensus.cc:348] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.949556 9870 raft_consensus.cc:374] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:23.949595 9870 raft_consensus.cc:729] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9891ab96fbe1479492a10687489a04fa, State: Initialized, Role: FOLLOWER
I20260928 14:41:23.949728 9870 consensus_queue.cc:260] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.949719 9869 raft_consensus.cc:348] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.949872 9869 raft_consensus.cc:374] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:23.949925 9869 raft_consensus.cc:729] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6762c03b2039486fb9ecb50b070a7b64, State: Initialized, Role: FOLLOWER
I20260928 14:41:23.950037 9870 ts_tablet_manager.cc:1428] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:23.950047 9869 consensus_queue.cc:260] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.950201 9868 tablet_bootstrap.cc:492] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a: No bootstrap required, opened a new log
I20260928 14:41:23.950273 9868 ts_tablet_manager.cc:1397] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:23.950315 9869 ts_tablet_manager.cc:1428] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:23.950639 9868 raft_consensus.cc:348] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.950744 9868 raft_consensus.cc:374] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:23.950786 9868 raft_consensus.cc:729] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7216319cfbe94d438e63a2b14976f23a, State: Initialized, Role: FOLLOWER
I20260928 14:41:23.950904 9868 consensus_queue.cc:260] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:23.951161 9868 ts_tablet_manager.cc:1428] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:41:24.158488 9913 raft_consensus.cc:482] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:41:24.158587 9913 raft_consensus.cc:504] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:24.159047 9913 leader_election.cc:288] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059), 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607)
I20260928 14:41:24.159399 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d87a9fc438d4ebc83956df3ae601d27" candidate_uuid: "9891ab96fbe1479492a10687489a04fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6762c03b2039486fb9ecb50b070a7b64" is_pre_election: true
I20260928 14:41:24.159441 9794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d87a9fc438d4ebc83956df3ae601d27" candidate_uuid: "9891ab96fbe1479492a10687489a04fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7216319cfbe94d438e63a2b14976f23a" is_pre_election: true
I20260928 14:41:24.159548 9654 raft_consensus.cc:2454] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9891ab96fbe1479492a10687489a04fa in term 0.
I20260928 14:41:24.159562 9794 raft_consensus.cc:2454] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9891ab96fbe1479492a10687489a04fa in term 0.
I20260928 14:41:24.159857 9448 leader_election.cc:302] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6762c03b2039486fb9ecb50b070a7b64, 7216319cfbe94d438e63a2b14976f23a, 9891ab96fbe1479492a10687489a04fa; no voters:
I20260928 14:41:24.160059 9913 raft_consensus.cc:2781] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:41:24.160125 9913 raft_consensus.cc:482] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:41:24.160151 9913 raft_consensus.cc:3037] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:24.161087 9913 raft_consensus.cc:504] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:24.161379 9913 leader_election.cc:288] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [CANDIDATE]: Term 1 election: Requested vote from peers 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059), 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607)
I20260928 14:41:24.161665 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d87a9fc438d4ebc83956df3ae601d27" candidate_uuid: "9891ab96fbe1479492a10687489a04fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6762c03b2039486fb9ecb50b070a7b64"
I20260928 14:41:24.161665 9794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d87a9fc438d4ebc83956df3ae601d27" candidate_uuid: "9891ab96fbe1479492a10687489a04fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7216319cfbe94d438e63a2b14976f23a"
I20260928 14:41:24.161768 9654 raft_consensus.cc:3037] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:24.161768 9794 raft_consensus.cc:3037] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:24.162776 9654 raft_consensus.cc:2454] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9891ab96fbe1479492a10687489a04fa in term 1.
I20260928 14:41:24.162776 9794 raft_consensus.cc:2454] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9891ab96fbe1479492a10687489a04fa in term 1.
I20260928 14:41:24.163014 9448 leader_election.cc:302] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7216319cfbe94d438e63a2b14976f23a, 9891ab96fbe1479492a10687489a04fa; no voters:
I20260928 14:41:24.163166 9913 raft_consensus.cc:2781] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:24.163244 9913 raft_consensus.cc:686] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 1 LEADER]: Becoming Leader. State: Replica: 9891ab96fbe1479492a10687489a04fa, State: Running, Role: LEADER
I20260928 14:41:24.163378 9913 consensus_queue.cc:237] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } }
I20260928 14:41:24.164846 9356 catalog_manager.cc:5495] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9891ab96fbe1479492a10687489a04fa (127.1.201.1). New cstate: current_term: 1 leader_uuid: "9891ab96fbe1479492a10687489a04fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891ab96fbe1479492a10687489a04fa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 41257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 } health_report { overall_health: UNKNOWN } } }
I20260928 14:41:24.184439 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:24.181631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.1.201.1:41257)
0928 14:41:24.181768 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:24.181772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:24.181800 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:41:24.181881 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:24.181884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:24.181929 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:24.182000 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:24.182225 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:24.182740 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:24.183502 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:24.183719 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:24.183821 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:24.184046 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:24.184114 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:24.184186 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:24.184271 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 14:41:24.184314 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 14:41:24.191780 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:24.189294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:36191 (local address 127.1.201.1:41257)
0928 14:41:24.189391 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:24.189394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:24.189405 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:41:24.189474 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:24.189476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:24.189511 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:24.189566 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:24.189730 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:24.190252 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:24.190954 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:24.191138 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:24.191214 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:24.191433 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:24.191491 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:24.191539 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:24.191610 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 14:41:24.191672 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":36}
I20260928 14:41:24.191797 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:24.189209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:41257 (local address 127.1.201.1:36191)
0928 14:41:24.189336 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:24.189357 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:24.189416 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:24.189585 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:24.189587 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:24.189593 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:41:24.189664 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:24.189668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:24.190332 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:24.190334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:24.190862 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:24.190869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:24.191230 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:24.191232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:24.191297 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:24.191317 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:41:24.191555 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:41:24.191559 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:41:24.191606 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 14:41:24.191698 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":16}
I20260928 14:41:24.196632 9794 raft_consensus.cc:1261] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Refusing update from remote peer 9891ab96fbe1479492a10687489a04fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:24.196631 9654 raft_consensus.cc:1261] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Refusing update from remote peer 9891ab96fbe1479492a10687489a04fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:24.196940 9913 consensus_queue.cc:1035] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7216319cfbe94d438e63a2b14976f23a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:24.197075 9939 consensus_queue.cc:1035] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6762c03b2039486fb9ecb50b070a7b64" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:24.226521 9357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:24.226640 9357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:24.228492 9357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b14ab93a1e54af7a6fb186a8ef130bf: Sending DeleteTablet for 3 replicas of tablet 0d87a9fc438d4ebc83956df3ae601d27
I20260928 14:41:24.229003 9633 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d87a9fc438d4ebc83956df3ae601d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:24 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:37078
I20260928 14:41:24.229003 9493 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d87a9fc438d4ebc83956df3ae601d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:24 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:49520
I20260928 14:41:24.229074 9774 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d87a9fc438d4ebc83956df3ae601d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:24 UTC) from {username='oryx', principal='oryx/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:55248
I20260928 14:41:24.229167 9941 tablet_replica.cc:330] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64: stopping tablet replica
I20260928 14:41:24.229245 9943 tablet_replica.cc:330] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a: stopping tablet replica
I20260928 14:41:24.229288 9941 raft_consensus.cc:2229] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:24.229342 9943 raft_consensus.cc:2229] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:24.229385 9941 raft_consensus.cc:2258] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:24.229431 9943 raft_consensus.cc:2258] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:24.229535 9942 tablet_replica.cc:330] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa: stopping tablet replica
I20260928 14:41:24.229661 9942 raft_consensus.cc:2229] T 0d87a9fc438d4ebc83956df3ae601d27 P 9891ab96fbe1479492a10687489a04fa [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:41:24.230109 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 9429
I20260928 14:41:24.230140 9943 ts_tablet_manager.cc:1905] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:24.231389 9941 ts_tablet_manager.cc:1905] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:24.232677 9943 ts_tablet_manager.cc:1918] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:41:24.232758 9943 log.cc:1196] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/0d87a9fc438d4ebc83956df3ae601d27
I20260928 14:41:24.233016 9943 ts_tablet_manager.cc:1939] T 0d87a9fc438d4ebc83956df3ae601d27 P 7216319cfbe94d438e63a2b14976f23a: Deleting consensus metadata
I20260928 14:41:24.233994 9941 ts_tablet_manager.cc:1918] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:41:24.234069 9941 log.cc:1196] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/0d87a9fc438d4ebc83956df3ae601d27
I20260928 14:41:24.234079 9344 catalog_manager.cc:4841] TS 7216319cfbe94d438e63a2b14976f23a (127.1.201.3:43607): tablet 0d87a9fc438d4ebc83956df3ae601d27 (table test-table [id=312d62c89de4488eb9b230e6366ae45d]) successfully deleted
I20260928 14:41:24.234316 9941 ts_tablet_manager.cc:1939] T 0d87a9fc438d4ebc83956df3ae601d27 P 6762c03b2039486fb9ecb50b070a7b64: Deleting consensus metadata
I20260928 14:41:24.234889 9343 catalog_manager.cc:4841] TS 6762c03b2039486fb9ecb50b070a7b64 (127.1.201.2:34059): tablet 0d87a9fc438d4ebc83956df3ae601d27 (table test-table [id=312d62c89de4488eb9b230e6366ae45d]) successfully deleted
W20260928 14:41:24.239442 9342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:41257 (error 108)
I20260928 14:41:24.239668 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 9569
I20260928 14:41:24.240773 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:24.240362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:41257 (local address 127.0.0.1:49556)
0928 14:41:24.240576 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:24.240648 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:41257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":55,"threads_started":1}
W20260928 14:41:24.240916 9342 catalog_manager.cc:4584] TS 9891ab96fbe1479492a10687489a04fa (127.1.201.1:41257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d87a9fc438d4ebc83956df3ae601d27: Network error: Client connection negotiation failed: client connection to 127.1.201.1:41257: connect: Connection refused (error 111)
I20260928 14:41:24.246081 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 9709
I20260928 14:41:24.252097 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 9325
2026-09-28T14:41:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3656 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:41:24 dist-test-slave-0n27 krb5kdc[9975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:41:24 dist-test-slave-0n27 krb5kdc[9975](info): set up 2 sockets
Sep 28 14:41:24 dist-test-slave-0n27 krb5kdc[9975](info): commencing operation
krb5kdc: starting...
W20260928 14:41:25.231290 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:41:25 dist-test-slave-0n27 krb5kdc[9975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:41:25.340728 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:45017
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:45017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:25.442127 9988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:25.442363 9988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:25.442432 9988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:25.446312 9988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:25.446357 9988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:25.446373 9988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:25.446422 9988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:25.446446 9988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:25.451277 9988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:45017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:45017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:25.452099 9988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:25.452989 9988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:25.460268 9994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:25.460282 9996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:25.460459 9988 server_base.cc:1034] running on GCE node
W20260928 14:41:25.460280 9993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:25.461028 9988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:25.462071 9988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:25.463269 9988 hybrid_clock.cc:648] HybridClock initialized: now 1790606485463246 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T14:41:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1224 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:41:25 dist-test-slave-0n27 krb5kdc[10001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:41:25 dist-test-slave-0n27 krb5kdc[10001](info): set up 2 sockets
Sep 28 14:41:25 dist-test-slave-0n27 krb5kdc[10001](info): commencing operation
krb5kdc: starting...
W20260928 14:41:26.455157 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:41:26 dist-test-slave-0n27 krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:41:26.580152 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:33287
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:33287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:26.683184 10014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:26.683418 10014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:26.683454 10014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:26.687247 10014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:26.687292 10014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:26.687311 10014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:26.687326 10014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:26.687390 10014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:26.692421 10014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:33287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:33287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:26.693353 10014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:26.694358 10014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:26.702472 10019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:26.702569 10014 server_base.cc:1034] running on GCE node
W20260928 14:41:26.702490 10020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:26.702504 10022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:26.703115 10014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:26.704109 10014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:26.705294 10014 hybrid_clock.cc:648] HybridClock initialized: now 1790606486705268 us; error 39 us; skew 500 ppm
Sep 28 14:41:26 dist-test-slave-0n27 krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606486, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:26.707845 10014 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:41:26.708639 10014 webserver.cc:460] Webserver started at http://127.1.201.62:37899/ using document root <none> and password file <none>
I20260928 14:41:26.709156 10014 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:26.709213 10014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:26.709367 10014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:26.711313 10014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "bcd4a2a8168d45db957801880f80f226"
format_stamp: "Formatted at 2026-09-28 14:41:26 on dist-test-slave-0n27"
server_key: "47de98772030e2f2bd045f0f4e4d8643"
server_key_iv: "2aac02fbcb8a9ac233bb4a00646e1fb9"
server_key_version: "encryptionkey@0"
I20260928 14:41:26.711810 10014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "bcd4a2a8168d45db957801880f80f226"
format_stamp: "Formatted at 2026-09-28 14:41:26 on dist-test-slave-0n27"
server_key: "47de98772030e2f2bd045f0f4e4d8643"
server_key_iv: "2aac02fbcb8a9ac233bb4a00646e1fb9"
server_key_version: "encryptionkey@0"
I20260928 14:41:26.715261 10014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 14:41:26.717480 10029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:26.718622 10014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:26.718729 10014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "bcd4a2a8168d45db957801880f80f226"
format_stamp: "Formatted at 2026-09-28 14:41:26 on dist-test-slave-0n27"
server_key: "47de98772030e2f2bd045f0f4e4d8643"
server_key_iv: "2aac02fbcb8a9ac233bb4a00646e1fb9"
server_key_version: "encryptionkey@0"
I20260928 14:41:26.718806 10014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:26.746119 10014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:26.748848 10014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:26.749001 10014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:26.756027 10014 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:33287
I20260928 14:41:26.756042 10081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:33287 every 8 connection(s)
I20260928 14:41:26.756958 10014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:26.759586 10082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:26.764945 10082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226: Bootstrap starting.
I20260928 14:41:26.765817 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 10014
I20260928 14:41:26.765960 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:26.766213 1828 external_mini_cluster.cc:1428] Setting key 6df4b25d0a1ac8d8972e75256467ac69
I20260928 14:41:26.767640 10082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:26.768419 10082 log.cc:824] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:26.770535 10082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226: No bootstrap required, opened a new log
I20260928 14:41:26.774840 10082 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } }
Sep 28 14:41:26 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:26.775059 10082 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:26.775147 10082 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd4a2a8168d45db957801880f80f226, State: Initialized, Role: FOLLOWER
I20260928 14:41:26.775535 10082 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } }
I20260928 14:41:26.775709 10082 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:26.775794 10082 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:26.775908 10082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:26.776949 10082 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } }
I20260928 14:41:26.777266 10082 leader_election.cc:302] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcd4a2a8168d45db957801880f80f226; no voters:
I20260928 14:41:26.777565 10082 leader_election.cc:288] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:26.777635 10087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:26.777841 10087 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 1 LEADER]: Becoming Leader. State: Replica: bcd4a2a8168d45db957801880f80f226, State: Running, Role: LEADER
I20260928 14:41:26.778148 10087 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } }
I20260928 14:41:26.778723 10082 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:26.779778 10088 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcd4a2a8168d45db957801880f80f226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } } }
I20260928 14:41:26.779896 10088 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:26.780277 10089 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcd4a2a8168d45db957801880f80f226. Latest consensus state: current_term: 1 leader_uuid: "bcd4a2a8168d45db957801880f80f226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } } }
I20260928 14:41:26.780426 10089 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:26.780555 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:26.767288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.1.201.62:33287)
0928 14:41:26.767689 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:26.767704 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:26.767757 (+ 53us) server_negotiation.cc:407] Connection header received
0928 14:41:26.768419 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:26.768434 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:26.768860 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:26.769217 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:26.770150 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:26.771140 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:26.772331 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:26.772749 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:26.772852 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:26.775683 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:26.775687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:26.775720 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:26.775749 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:26.777412 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:26.777955 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:26.777960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:26.777965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:26.778079 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:26.778379 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:26.778382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:26.778383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:26.778729 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:26.778940 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:26.779371 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 14:41:26.779652 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":117,"threads_started":1}
I20260928 14:41:26.780824 10096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:26.783777 10096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:26.788537 10096 catalog_manager.cc:1347] Generated new cluster ID: 31faf4509cd0463b95d1828ab4a6d674
I20260928 14:41:26.788594 10096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:26.803541 10096 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:26.804397 10096 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:26.808771 10096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226: Generated new TSK 0
I20260928 14:41:26.809405 10096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:41:26.840004 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:33287
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:26.961148 10108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:26.961360 10108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:26.961397 10108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:26.965101 10108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:26.965148 10108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:26.965197 10108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:26.970170 10108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:26.971052 10108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:26.971921 10108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:26.980844 10114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:26.980872 10113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:26.980917 10108 server_base.cc:1034] running on GCE node
W20260928 14:41:26.980973 10116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:26.981372 10108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:26.982045 10108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:26.983242 10108 hybrid_clock.cc:648] HybridClock initialized: now 1790606486983211 us; error 43 us; skew 500 ppm
Sep 28 14:41:26 dist-test-slave-0n27 krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606486, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:26.985844 10108 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:41:26.986579 10108 webserver.cc:460] Webserver started at http://127.1.201.1:37821/ using document root <none> and password file <none>
I20260928 14:41:26.987092 10108 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:26.987200 10108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:26.987401 10108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:26.989212 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "8775e698c7004bb09441ff5f243edd73"
format_stamp: "Formatted at 2026-09-28 14:41:26 on dist-test-slave-0n27"
server_key: "c834523fcb35b52d1b9f08b3f86a4830"
server_key_iv: "a22ffabc9d0f55e1280d3292fec54bf3"
server_key_version: "encryptionkey@0"
I20260928 14:41:26.989740 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "8775e698c7004bb09441ff5f243edd73"
format_stamp: "Formatted at 2026-09-28 14:41:26 on dist-test-slave-0n27"
server_key: "c834523fcb35b52d1b9f08b3f86a4830"
server_key_iv: "a22ffabc9d0f55e1280d3292fec54bf3"
server_key_version: "encryptionkey@0"
I20260928 14:41:26.993131 10108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:41:26.995347 10123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:26.996392 10108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:26.996517 10108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "8775e698c7004bb09441ff5f243edd73"
format_stamp: "Formatted at 2026-09-28 14:41:26 on dist-test-slave-0n27"
server_key: "c834523fcb35b52d1b9f08b3f86a4830"
server_key_iv: "a22ffabc9d0f55e1280d3292fec54bf3"
server_key_version: "encryptionkey@0"
I20260928 14:41:26.996618 10108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:27.006096 10108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:27.008833 10108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:27.009047 10108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:27.009545 10108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:27.010309 10108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:27.010401 10108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.010463 10108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:27.010505 10108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.018262 10108 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:44475
I20260928 14:41:27.018281 10236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:44475 every 8 connection(s)
I20260928 14:41:27.019019 10108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:27.025875 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 10108
I20260928 14:41:27.025990 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:27.026294 1828 external_mini_cluster.cc:1428] Setting key e21e7815e11f9f0731b52299d240621a
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606486, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:27.032449 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:27.020645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:51221 (local address 127.1.201.62:33287)
0928 14:41:27.020787 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:27.020791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:27.021468 (+ 677us) server_negotiation.cc:407] Connection header received
0928 14:41:27.022702 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:27.022705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:27.022759 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:27.022849 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:27.023890 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.024421 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.025446 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.025680 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.025783 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.028590 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:27.028593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:27.028595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:27.028615 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:27.030399 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:27.030881 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:27.030887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:27.030891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:27.030987 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:27.031306 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:27.031311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:27.031316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:27.031559 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:27.031660 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:27.032094 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 14:41:27.032235 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:41:27.033085 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:27.020926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:51221)
0928 14:41:27.021334 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:27.021384 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:27.022510 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:27.022996 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:27.023002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:27.023300 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:27.023718 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.023754 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.024540 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.024543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.025331 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.025339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.025835 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.025840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.026066 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.026702 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:27.026717 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:27.028416 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:27.030513 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:27.030517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:27.030527 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:27.030769 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:27.031076 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:27.031078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:27.031080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:27.031216 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:27.031663 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:27.031668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:27.031869 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 14:41:27.032007 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 14:41:27.032239 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":91,"threads_started":1}
I20260928 14:41:27.034258 10237 heartbeater.cc:347] Connected to a master server at 127.1.201.62:33287
I20260928 14:41:27.034508 10237 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:27.034948 10237 heartbeater.cc:511] Master 127.1.201.62:33287 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
I20260928 14:41:27.036441 10046 ts_manager.cc:194] Registered new tserver with Master: 8775e698c7004bb09441ff5f243edd73 (127.1.201.1:44475)
I20260928 14:41:27.037665 10046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:51221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:41:27.053208 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:33287
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:27.156145 10242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:27.156379 10242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:27.156438 10242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:27.160275 10242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:27.160344 10242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:27.160429 10242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:27.164979 10242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:27.165863 10242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:27.166685 10242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:27.175001 10250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:27.175060 10248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:27.175076 10247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:27.175272 10242 server_base.cc:1034] running on GCE node
I20260928 14:41:27.175776 10242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:27.176285 10242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:27.177440 10242 hybrid_clock.cc:648] HybridClock initialized: now 1790606487177415 us; error 36 us; skew 500 ppm
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:27.180315 10242 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:41:27.181035 10242 webserver.cc:460] Webserver started at http://127.1.201.2:35519/ using document root <none> and password file <none>
I20260928 14:41:27.181501 10242 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:27.181555 10242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:27.181708 10242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:27.183784 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "0e930a0ecd4b4345bbddc56dff1e5988"
format_stamp: "Formatted at 2026-09-28 14:41:27 on dist-test-slave-0n27"
server_key: "973166c823dd10a5da55c114031ba5a4"
server_key_iv: "c139cef0d4418548fad5a87c73b1d3f7"
server_key_version: "encryptionkey@0"
I20260928 14:41:27.184243 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "0e930a0ecd4b4345bbddc56dff1e5988"
format_stamp: "Formatted at 2026-09-28 14:41:27 on dist-test-slave-0n27"
server_key: "973166c823dd10a5da55c114031ba5a4"
server_key_iv: "c139cef0d4418548fad5a87c73b1d3f7"
server_key_version: "encryptionkey@0"
I20260928 14:41:27.187533 10242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:27.189687 10257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.190820 10242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:27.190902 10242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "0e930a0ecd4b4345bbddc56dff1e5988"
format_stamp: "Formatted at 2026-09-28 14:41:27 on dist-test-slave-0n27"
server_key: "973166c823dd10a5da55c114031ba5a4"
server_key_iv: "c139cef0d4418548fad5a87c73b1d3f7"
server_key_version: "encryptionkey@0"
I20260928 14:41:27.190974 10242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:27.202145 10242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:27.204553 10242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:27.204706 10242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:27.205164 10242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:27.205922 10242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:27.205965 10242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.206000 10242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:27.206018 10242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.214087 10242 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:34139
I20260928 14:41:27.214113 10370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:34139 every 8 connection(s)
I20260928 14:41:27.214844 10242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:27.218299 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 10242
I20260928 14:41:27.218431 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:27.218726 1828 external_mini_cluster.cc:1428] Setting key bd1b4ce209f73a8ff07feb3e29318f8e
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
I20260928 14:41:27.228142 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:27.216373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:38809 (local address 127.1.201.62:33287)
0928 14:41:27.216496 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:27.216499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:27.217199 (+ 700us) server_negotiation.cc:407] Connection header received
0928 14:41:27.218737 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:27.218740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:27.218813 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:27.218915 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:27.220236 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.220720 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.221818 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.222116 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.222216 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.224815 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:27.224817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:27.224819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:27.224840 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:27.226174 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:27.226735 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:27.226737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:27.226738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:27.226802 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:27.227143 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:27.227145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:27.227146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:27.227286 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:27.227351 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:27.227909 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 14:41:27.228036 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:41:27.228952 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:27.216619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:38809)
0928 14:41:27.217055 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:27.217093 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:27.218513 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:27.219101 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:27.219107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:27.219546 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:27.220079 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.220094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.220832 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.220834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.221684 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.221693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.222247 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.222251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.222449 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.222986 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:27.223001 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:27.224651 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:27.226282 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:27.226288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:27.226301 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:27.226629 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:27.226886 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:27.226891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:27.226895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:27.227052 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:27.227387 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:27.227395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:27.227689 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 14:41:27.227882 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 14:41:27.228189 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260928 14:41:27.230185 10371 heartbeater.cc:347] Connected to a master server at 127.1.201.62:33287
I20260928 14:41:27.230425 10371 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
I20260928 14:41:27.230952 10371 heartbeater.cc:511] Master 127.1.201.62:33287 requested a full tablet report, sending...
I20260928 14:41:27.231954 10046 ts_manager.cc:194] Registered new tserver with Master: 0e930a0ecd4b4345bbddc56dff1e5988 (127.1.201.2:34139)
I20260928 14:41:27.232703 10046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:38809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:41:27.245163 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:33287
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:27.346930 10376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:27.347121 10376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:27.347157 10376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:27.350958 10376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:27.351001 10376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:27.351049 10376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:27.355650 10376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:27.356400 10376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:27.357156 10376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:27.365880 10384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:27.365964 10376 server_base.cc:1034] running on GCE node
W20260928 14:41:27.365867 10381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:27.365869 10382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:27.366566 10376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:27.367123 10376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:27.368271 10376 hybrid_clock.cc:648] HybridClock initialized: now 1790606487368246 us; error 37 us; skew 500 ppm
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:27.371055 10376 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:41:27.371941 10376 webserver.cc:460] Webserver started at http://127.1.201.3:35545/ using document root <none> and password file <none>
I20260928 14:41:27.372462 10376 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:27.372541 10376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:27.372751 10376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:27.374603 10376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "fed3edc899a548928960b76c8fc66d22"
format_stamp: "Formatted at 2026-09-28 14:41:27 on dist-test-slave-0n27"
server_key: "ba83f1c64ac8d96594bd5658fdc022d2"
server_key_iv: "b049ec41c1a38a3efe107cd0cfbe8a9f"
server_key_version: "encryptionkey@0"
I20260928 14:41:27.375109 10376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "fed3edc899a548928960b76c8fc66d22"
format_stamp: "Formatted at 2026-09-28 14:41:27 on dist-test-slave-0n27"
server_key: "ba83f1c64ac8d96594bd5658fdc022d2"
server_key_iv: "b049ec41c1a38a3efe107cd0cfbe8a9f"
server_key_version: "encryptionkey@0"
I20260928 14:41:27.378574 10376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:41:27.380857 10391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.382033 10376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:27.382149 10376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "fed3edc899a548928960b76c8fc66d22"
format_stamp: "Formatted at 2026-09-28 14:41:27 on dist-test-slave-0n27"
server_key: "ba83f1c64ac8d96594bd5658fdc022d2"
server_key_iv: "b049ec41c1a38a3efe107cd0cfbe8a9f"
server_key_version: "encryptionkey@0"
I20260928 14:41:27.382249 10376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:27.404209 10376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:27.407169 10376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:27.407367 10376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:27.407882 10376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:27.408627 10376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:27.408700 10376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.408787 10376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:27.408828 10376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.416996 10376 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:39325
I20260928 14:41:27.417038 10504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:39325 every 8 connection(s)
I20260928 14:41:27.417791 10376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:41:27.420415 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 10376
I20260928 14:41:27.420522 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:27.420800 1828 external_mini_cluster.cc:1428] Setting key 90a9dbec60e2f34fbe977c72d7ea08f8
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:27.429860 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:27.419401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:38247 (local address 127.1.201.62:33287)
0928 14:41:27.419526 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:27.419529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:27.420275 (+ 746us) server_negotiation.cc:407] Connection header received
0928 14:41:27.421522 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:27.421524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:27.421577 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:27.421647 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:27.422891 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.423383 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.424349 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.424557 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.424686 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.426926 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:27.426929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:27.426931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:27.426951 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:27.428211 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:27.428623 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:27.428625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:27.428626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:27.428693 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:27.428959 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:27.428961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:27.428962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:27.429130 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:27.429206 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:27.429598 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 14:41:27.429686 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:41:27.430620 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:27.419732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:38247)
0928 14:41:27.420122 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:27.420161 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:27.421310 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:27.421866 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:27.421874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:27.422223 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:27.422760 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.422775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.423475 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.423477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.424248 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.424256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.424735 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.424739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.424893 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.425370 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:27.425384 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:27.426794 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:27.428301 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:27.428305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:27.428314 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:27.428526 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:27.428767 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:27.428769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:27.428770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:27.428884 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:27.429221 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:27.429225 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:27.429416 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 14:41:27.429554 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 14:41:27.429769 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20260928 14:41:27.431659 10505 heartbeater.cc:347] Connected to a master server at 127.1.201.62:33287
I20260928 14:41:27.431878 10505 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:27.432323 10505 heartbeater.cc:511] Master 127.1.201.62:33287 requested a full tablet report, sending...
I20260928 14:41:27.433391 10046 ts_manager.cc:194] Registered new tserver with Master: fed3edc899a548928960b76c8fc66d22 (127.1.201.3:39325)
I20260928 14:41:27.434103 10046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:38247
I20260928 14:41:27.434463 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab.
Entry for principal oryx/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab.
WARNING: no policy specified for oryx/127.1.201.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab.
Entry for principal oryx/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab.
WARNING: no policy specified for oryx/127.1.201.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab.
Entry for principal oryx/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab.
I20260928 14:41:27.498370 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 10108
I20260928 14:41:27.503944 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 10242
I20260928 14:41:27.508882 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 10376
I20260928 14:41:27.513660 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 10014
I20260928 14:41:27.519479 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:33287
--webserver_interface=127.1.201.62
--webserver_port=37899
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:33287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:27.624687 10514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:27.624905 10514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:27.624970 10514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:27.628836 10514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:27.628902 10514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:27.628921 10514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:27.628937 10514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:27.628954 10514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:27.633695 10514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:33287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:33287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:27.634537 10514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:27.635346 10514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:27.642963 10520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:27.642973 10522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:27.642967 10519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:27.643203 10514 server_base.cc:1034] running on GCE node
I20260928 14:41:27.643797 10514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:27.644802 10514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:27.645980 10514 hybrid_clock.cc:648] HybridClock initialized: now 1790606487645961 us; error 35 us; skew 500 ppm
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:27.648538 10514 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:41:27.649344 10514 webserver.cc:460] Webserver started at http://127.1.201.62:37899/ using document root <none> and password file <none>
I20260928 14:41:27.649911 10514 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:27.650010 10514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:27.653359 10514 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:27.655292 10529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.656340 10514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:27.656476 10514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "bcd4a2a8168d45db957801880f80f226"
format_stamp: "Formatted at 2026-09-28 14:41:26 on dist-test-slave-0n27"
server_key: "47de98772030e2f2bd045f0f4e4d8643"
server_key_iv: "2aac02fbcb8a9ac233bb4a00646e1fb9"
server_key_version: "encryptionkey@0"
I20260928 14:41:27.656944 10514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:27.674767 10514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:27.677688 10514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:27.677893 10514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:27.684267 10514 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:33287
I20260928 14:41:27.684291 10581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:33287 every 8 connection(s)
I20260928 14:41:27.685187 10514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:27.689019 10582 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:41:27.690606 10582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226: Bootstrap starting.
I20260928 14:41:27.695119 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 10514
I20260928 14:41:27.695699 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:44475
--local_ip_for_outbound_sockets=127.1.201.1
--tserver_master_addrs=127.1.201.62:33287
--webserver_port=37821
--webserver_interface=127.1.201.1
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.201.1
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:41:27.707315 10582 log.cc:824] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:27.713559 10582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:41:27.713955 10582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226: Bootstrap complete.
I20260928 14:41:27.718012 10582 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } }
I20260928 14:41:27.718389 10582 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd4a2a8168d45db957801880f80f226, State: Initialized, Role: FOLLOWER
I20260928 14:41:27.718909 10582 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } }
I20260928 14:41:27.719044 10582 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:27.719133 10582 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:27.719250 10582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:41:27.721158 10582 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } }
I20260928 14:41:27.721485 10582 leader_election.cc:302] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcd4a2a8168d45db957801880f80f226; no voters:
I20260928 14:41:27.721824 10582 leader_election.cc:288] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:41:27.721900 10586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:41:27.722121 10586 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [term 2 LEADER]: Becoming Leader. State: Replica: bcd4a2a8168d45db957801880f80f226, State: Running, Role: LEADER
I20260928 14:41:27.722404 10586 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } }
I20260928 14:41:27.723024 10582 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:27.723397 10588 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcd4a2a8168d45db957801880f80f226. Latest consensus state: current_term: 2 leader_uuid: "bcd4a2a8168d45db957801880f80f226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } } }
I20260928 14:41:27.723507 10588 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:27.723742 10587 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bcd4a2a8168d45db957801880f80f226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd4a2a8168d45db957801880f80f226" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 33287 } } }
I20260928 14:41:27.723834 10587 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:27.723875 10590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:27.727108 10590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:27.727738 10590 catalog_manager.cc:1259] Loaded cluster ID: 31faf4509cd0463b95d1828ab4a6d674
I20260928 14:41:27.733987 10590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:27.736222 10590 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:27.736866 10590 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bcd4a2a8168d45db957801880f80f226: Loaded TSK: 0
I20260928 14:41:27.737957 10590 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:41:27.804657 10584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:27.804888 10584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:27.804922 10584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:27.808626 10584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:27.808677 10584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:27.808764 10584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:27.813552 10584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.1.keytab
--principal=oryx/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:44475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=37821
--tserver_master_addrs=127.1.201.62:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:27.814433 10584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:27.815318 10584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:27.823943 10609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:27.823956 10608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:27.823956 10611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:27.824671 10584 server_base.cc:1034] running on GCE node
I20260928 14:41:27.825156 10584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:27.825980 10584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:27.827186 10584 hybrid_clock.cc:648] HybridClock initialized: now 1790606487827162 us; error 42 us; skew 500 ppm
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606487, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:27.830353 10584 init.cc:377] Logged in from keytab as oryx/127.1.201.1@KRBTEST.COM (short username oryx)
I20260928 14:41:27.831184 10584 webserver.cc:460] Webserver started at http://127.1.201.1:37821/ using document root <none> and password file <none>
I20260928 14:41:27.831689 10584 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:27.831758 10584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:27.835558 10584 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:27.837620 10618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.838841 10584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:27.838977 10584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "8775e698c7004bb09441ff5f243edd73"
format_stamp: "Formatted at 2026-09-28 14:41:26 on dist-test-slave-0n27"
server_key: "c834523fcb35b52d1b9f08b3f86a4830"
server_key_iv: "a22ffabc9d0f55e1280d3292fec54bf3"
server_key_version: "encryptionkey@0"
I20260928 14:41:27.839605 10584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:27.877914 10584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:27.881067 10584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:27.881255 10584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:27.881841 10584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:27.882616 10584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:27.882660 10584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.882733 10584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:27.882762 10584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:27.891366 10584 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:44475
I20260928 14:41:27.891400 10731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:44475 every 8 connection(s)
I20260928 14:41:27.892277 10584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:27.895707 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 10584
I20260928 14:41:27.896107 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:34139
--local_ip_for_outbound_sockets=127.1.201.2
--tserver_master_addrs=127.1.201.62:33287
--webserver_port=35519
--webserver_interface=127.1.201.2
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.201.2
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:27.905226 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:27.894362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:45765)
0928 14:41:27.894774 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:27.894834 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:27.896016 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:27.897243 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:27.897248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:27.897593 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:27.898100 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.898114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.899715 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.899717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.900536 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.900545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.901347 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.901351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.901522 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.902086 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:27.902102 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:27.904611 (+ 2509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":129,"threads_started":1}
W20260928 14:41:27.905632 10732 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:33287 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.201.62:33287: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
W20260928 14:41:27.905746 10734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:27.894234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:45765 (local address 127.1.201.62:33287)
0928 14:41:27.894634 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:27.894654 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:27.894938 (+ 284us) server_negotiation.cc:407] Connection header received
0928 14:41:27.896418 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:27.896435 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:27.896845 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:27.897185 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:27.898268 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.899597 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.900663 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.901182 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.901362 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.904898 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:45765: BlockingRecv error: recv got EOF from 127.1.201.1:45765 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:27.911121 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:27.906017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:60333)
0928 14:41:27.906173 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:27.906209 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:27.906299 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:27.906577 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:27.906579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:27.906702 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:27.906780 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.906784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.907784 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.907786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.908245 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.908252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.908788 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.908790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.908848 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.909295 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:27.909308 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:27.911036 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260928 14:41:27.911280 10734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:27.906096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:60333 (local address 127.1.201.62:33287)
0928 14:41:27.906359 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:27.906366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:27.906382 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:41:27.906421 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:27.906427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:27.906480 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:27.906574 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:27.906898 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.907679 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.908349 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.908692 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.908787 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.911170 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:60333: BlockingRecv error: recv got EOF from 127.1.201.1:60333 (error 108)
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":180}
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:27 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:27.916453 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:27.911560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:49541)
0928 14:41:27.911828 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:27.911842 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:27.911946 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:27.912182 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:27.912184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:27.912345 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:27.912413 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.912418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.913078 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.913082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.913574 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:27.913581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.914109 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.914113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:27.914217 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.914644 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:27.914655 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:27.916307 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":108,"threads_started":1}
W20260928 14:41:27.916507 10734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:27.911895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:49541 (local address 127.1.201.62:33287)
0928 14:41:27.912016 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:27.912019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:27.912029 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:41:27.912056 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:27.912058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:27.912095 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:27.912167 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:27.912518 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.912981 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.913742 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:27.913992 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:27.914102 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:27.916410 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:49541: BlockingRecv error: recv got EOF from 127.1.201.1:49541 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 14:41:27.916626 10732 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:41:28.004794 10736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:28.005003 10736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:28.005041 10736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:28.008962 10736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:28.009016 10736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:28.009088 10736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:28.013900 10736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.2.keytab
--principal=oryx/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:34139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=35519
--tserver_master_addrs=127.1.201.62:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:28.014760 10736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:28.015717 10736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:28.024545 10743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:28.024581 10742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:28.024575 10745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:28.024840 10736 server_base.cc:1034] running on GCE node
I20260928 14:41:28.025306 10736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:28.026021 10736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:28.027184 10736 hybrid_clock.cc:648] HybridClock initialized: now 1790606488027156 us; error 40 us; skew 500 ppm
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606488, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:28.029664 10736 init.cc:377] Logged in from keytab as oryx/127.1.201.2@KRBTEST.COM (short username oryx)
I20260928 14:41:28.030505 10736 webserver.cc:460] Webserver started at http://127.1.201.2:35519/ using document root <none> and password file <none>
I20260928 14:41:28.030967 10736 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:28.031026 10736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:28.034348 10736 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 14:41:28.036139 10752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:28.037278 10736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:28.037400 10736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "0e930a0ecd4b4345bbddc56dff1e5988"
format_stamp: "Formatted at 2026-09-28 14:41:27 on dist-test-slave-0n27"
server_key: "973166c823dd10a5da55c114031ba5a4"
server_key_iv: "c139cef0d4418548fad5a87c73b1d3f7"
server_key_version: "encryptionkey@0"
I20260928 14:41:28.037943 10736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:28.048902 10736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:28.051510 10736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:28.051690 10736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:28.052222 10736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:28.052997 10736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:28.053041 10736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:28.053107 10736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:28.053161 10736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:28.062202 10736 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:34139
I20260928 14:41:28.062237 10865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:34139 every 8 connection(s)
I20260928 14:41:28.063073 10736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:28.063330 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 10736
I20260928 14:41:28.063900 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:39325
--local_ip_for_outbound_sockets=127.1.201.3
--tserver_master_addrs=127.1.201.62:33287
--webserver_port=35545
--webserver_interface=127.1.201.3
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.201.3
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:41:28.075821 10734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:28.065064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:41817 (local address 127.1.201.62:33287)
0928 14:41:28.065236 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:28.065243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:28.066134 (+ 891us) server_negotiation.cc:407] Connection header received
0928 14:41:28.067897 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:28.067900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:28.067958 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:28.068027 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:28.069616 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.070203 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.072004 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.072196 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.072264 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.075704 (+ 3440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:41817: BlockingRecv error: recv got EOF from 127.1.201.2:41817 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:41:28.076342 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:28.065424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:41817)
0928 14:41:28.065941 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:28.066000 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:28.067672 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:28.068253 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:28.068264 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:28.068747 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:28.069441 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.069464 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.070339 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.070344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.071886 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.071898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.072335 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.072341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.072550 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.073132 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:28.073157 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:28.075567 (+ 2410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
W20260928 14:41:28.076810 10866 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:33287 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.201.62:33287: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:28.082119 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:28.077285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:49617)
0928 14:41:28.077404 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:28.077418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:28.077486 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:28.077702 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:28.077704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:28.077867 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:28.077944 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.077949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.078874 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.078877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.079411 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.079418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.079950 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.079952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.080015 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.080482 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:28.080493 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:28.082044 (+ 1551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 14:41:28.082276 10734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:28.077367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:49617 (local address 127.1.201.62:33287)
0928 14:41:28.077464 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:28.077468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:28.077481 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:41:28.077556 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:28.077559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:28.077605 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:28.077668 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:28.078034 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.078773 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.079513 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.079853 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.079946 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.082174 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:49617: BlockingRecv error: recv got EOF from 127.1.201.2:49617 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:28.086988 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:28.082550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:36169)
0928 14:41:28.082680 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:28.082697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:28.082749 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:28.082933 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:28.082935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:28.083038 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:28.083094 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.083097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.083971 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.083972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.084437 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.084444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.084968 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.084970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.085022 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.085447 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:28.085457 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:28.086897 (+ 1440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 14:41:28.087137 10734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:28.082612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:36169 (local address 127.1.201.62:33287)
0928 14:41:28.082706 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:28.082709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:28.082722 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:41:28.082815 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:28.082818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:28.082850 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:28.082913 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:28.083191 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.083874 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.084539 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.084869 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.084966 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.087045 (+ 2079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:36169: BlockingRecv error: recv got EOF from 127.1.201.2:36169 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 14:41:28.087163 10866 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:41:28.172118 10868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:28.172322 10868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:28.172358 10868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:28.175941 10868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:28.175987 10868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:28.176035 10868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:28.180640 10868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/krb5kdc/oryx_127.1.201.3.keytab
--principal=oryx/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:39325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=35545
--tserver_master_addrs=127.1.201.62:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:28.181417 10868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:28.182281 10868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:28.191365 10875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:28.191506 10868 server_base.cc:1034] running on GCE node
W20260928 14:41:28.191401 10877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:28.191412 10874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:28.191990 10868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:28.192631 10868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:28.193838 10868 hybrid_clock.cc:648] HybridClock initialized: now 1790606488193825 us; error 24 us; skew 500 ppm
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606488, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:28.196403 10868 init.cc:377] Logged in from keytab as oryx/127.1.201.3@KRBTEST.COM (short username oryx)
I20260928 14:41:28.197288 10868 webserver.cc:460] Webserver started at http://127.1.201.3:35545/ using document root <none> and password file <none>
I20260928 14:41:28.197892 10868 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:28.198005 10868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:28.201749 10868 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:28.203707 10884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:28.204800 10868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:28.204941 10868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "fed3edc899a548928960b76c8fc66d22"
format_stamp: "Formatted at 2026-09-28 14:41:27 on dist-test-slave-0n27"
server_key: "ba83f1c64ac8d96594bd5658fdc022d2"
server_key_iv: "b049ec41c1a38a3efe107cd0cfbe8a9f"
server_key_version: "encryptionkey@0"
I20260928 14:41:28.205506 10868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:28.217336 10868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:28.219930 10868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:28.220111 10868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:28.220695 10868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:28.221524 10868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:28.221570 10868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:28.221630 10868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:28.221666 10868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:28.230962 10868 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:39325
I20260928 14:41:28.230993 10997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:39325 every 8 connection(s)
I20260928 14:41:28.231884 10868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:28.240808 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 10868
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:41:28.242285 10734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:28.233443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:51069 (local address 127.1.201.62:33287)
0928 14:41:28.233574 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:28.233578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:28.234424 (+ 846us) server_negotiation.cc:407] Connection header received
0928 14:41:28.235579 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:28.235581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:28.235639 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:28.235715 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:28.236765 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.237257 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.238365 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.238642 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.238736 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.242169 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:51069: BlockingRecv error: recv got EOF from 127.1.201.3:51069 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:41:28.242919 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:28.233882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:51069)
0928 14:41:28.234283 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:28.234322 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:28.235404 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:28.235869 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:28.235875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:28.236212 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:28.236645 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.236657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.237363 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.237366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.238267 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.238276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.238782 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.238785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.238910 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.239476 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:28.239492 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:28.242055 (+ 2563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
W20260928 14:41:28.243405 10998 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:33287 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.201.62:33287: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:28.248709 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:28.243743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:53365)
0928 14:41:28.243938 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:28.243947 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:28.244006 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:28.244185 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:28.244187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:28.244314 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:28.244385 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.244390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.245215 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.245218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.245729 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.245736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.246367 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.246369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.246430 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.246801 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:28.246817 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:28.248587 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140}
W20260928 14:41:28.248852 10734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:28.243838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:53365 (local address 127.1.201.62:33287)
0928 14:41:28.243947 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:28.243950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:28.243981 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:41:28.244069 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:28.244072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:28.244105 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:28.244171 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:28.244493 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.245110 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.245884 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.246261 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.246370 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.248730 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:53365: BlockingRecv error: recv got EOF from 127.1.201.3:53365 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:41:28.251045 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:28.241767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.1.201.62:33287)
0928 14:41:28.242023 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:28.242026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:28.242035 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:41:28.242061 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:28.242063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:28.242093 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:28.242198 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:28.242959 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.243725 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.244804 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.245012 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.245086 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.246659 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:28.246663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:28.246675 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:28.246722 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:28.248615 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:28.249132 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:28.249138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:28.249143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:28.249309 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:28.249802 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:28.249808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:28.249812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:28.250227 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:28.250423 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:28.250676 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 14:41:28.250873 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:28.254559 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:28.249337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:51823)
0928 14:41:28.249509 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:28.249524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:28.249593 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:28.249850 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:28.249852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:28.249966 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:28.250029 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.250033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.250971 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.250974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.251681 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.251690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.252101 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.252104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.252158 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.252539 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:28.252550 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:28.254442 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108}
W20260928 14:41:28.254632 10734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:28.249442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:51823 (local address 127.1.201.62:33287)
0928 14:41:28.249559 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:28.249562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:28.249570 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:41:28.249676 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:28.249678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:28.249706 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:28.249820 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:28.250182 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.250867 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.251791 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.251976 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.252091 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.254536 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:51823: BlockingRecv error: recv got EOF from 127.1.201.3:51823 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 14:41:28.254787 10998 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:28 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:28.922142 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:28.917202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:38303)
0928 14:41:28.917445 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:28.917458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:28.917533 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:28.917764 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:28.917766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:28.917941 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:28.918047 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.918052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.918718 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.918720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.919259 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:28.919267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.919657 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.919659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:28.919727 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.920179 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:28.920190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:28.921984 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
W20260928 14:41:28.922215 11004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:28.917279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:38303 (local address 127.1.201.62:33287)
0928 14:41:28.917504 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:28.917507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:28.917520 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:41:28.917616 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:28.917618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:28.917661 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:28.917731 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:28.918139 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.918630 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.919356 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:28.919566 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:28.919640 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:28.922091 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:38303: BlockingRecv error: recv got EOF from 127.1.201.1:38303 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
Sep 28 14:41:29 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:29 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:29.093240 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:29.087676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:59667)
0928 14:41:29.087958 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:29.087973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:29.088113 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:29.088371 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:29.088373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:29.088560 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:29.088643 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:29.088648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.089430 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.089435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:29.090085 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:29.090094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.090567 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.090570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:29.090648 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:29.091101 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:29.091111 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:29.093123 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
W20260928 14:41:29.093380 11004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:29.087798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:59667 (local address 127.1.201.62:33287)
0928 14:41:29.087953 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:29.087957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:29.088057 (+ 100us) server_negotiation.cc:407] Connection header received
0928 14:41:29.088219 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:29.088222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:29.088272 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:29.088349 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:29.088785 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.089322 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.090226 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.090463 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.090563 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:29.093252 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:59667: BlockingRecv error: recv got EOF from 127.1.201.2:59667 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 14:41:29 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:29 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:29.260277 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:29.255345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:55805)
0928 14:41:29.255580 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:29.255604 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:29.255691 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:29.255908 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:29.255910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:29.256088 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:29.256189 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:29.256195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.256887 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.256889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:29.257459 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:29.257465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.257892 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.257894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:29.257953 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:29.258360 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:29.258370 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:29.260146 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
W20260928 14:41:29.260427 11004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:29.255395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:55805 (local address 127.1.201.62:33287)
0928 14:41:29.255532 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:29.255535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:29.255668 (+ 133us) server_negotiation.cc:407] Connection header received
0928 14:41:29.255768 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:29.255770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:29.255814 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:29.255879 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:29.256285 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.256799 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.257551 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.257753 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.257880 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:29.260322 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:55805: BlockingRecv error: recv got EOF from 127.1.201.3:55805 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 14:41:29 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:29 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:29.927815 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:29.923015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:37839)
0928 14:41:29.923268 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:29.923280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:29.923369 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:29.923615 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:29.923617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:29.923782 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:29.923871 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:29.923875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.924692 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.924694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:29.925192 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:29.925198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.925589 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.925591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:29.925645 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:29.926102 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:29.926112 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:29.927690 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
W20260928 14:41:29.927927 11008 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:29.923067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:37839 (local address 127.1.201.62:33287)
0928 14:41:29.923288 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:29.923294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:29.923356 (+ 62us) server_negotiation.cc:407] Connection header received
0928 14:41:29.923453 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:29.923455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:29.923497 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:29.923605 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:29.923959 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.924564 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.925281 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:29.925504 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:29.925571 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:29.927808 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:37839: BlockingRecv error: recv got EOF from 127.1.201.1:37839 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":53,"threads_started":1}
Sep 28 14:41:30 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:30 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:30.099852 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:30.094234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:33345)
0928 14:41:30.094533 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:30.094546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:30.094677 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:30.094952 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:30.094955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:30.095151 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:30.095243 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:30.095248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.095979 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.095985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:30.096532 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:30.096540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.096982 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.096985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:30.097067 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:30.097585 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:30.097605 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:30.099639 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
W20260928 14:41:30.099920 11008 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:30.094346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:33345 (local address 127.1.201.62:33287)
0928 14:41:30.094497 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:30.094501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:30.094630 (+ 129us) server_negotiation.cc:407] Connection header received
0928 14:41:30.094795 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:30.094797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:30.094848 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:30.094932 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:30.095352 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.095873 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.096658 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.096877 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.096980 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:30.099782 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:33345: BlockingRecv error: recv got EOF from 127.1.201.2:33345 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 14:41:30 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:30 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:30.266088 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:30.261075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:44131)
0928 14:41:30.261348 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:30.261362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:30.261466 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:30.261733 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:30.261736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:30.261941 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:30.262028 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:30.262034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.262729 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.262733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:30.263253 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:30.263260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.263719 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.263723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:30.263805 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:30.264264 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:30.264275 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:30.265971 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":108,"threads_started":1}
W20260928 14:41:30.266245 11008 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:30.261209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:44131 (local address 127.1.201.62:33287)
0928 14:41:30.261344 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:30.261348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:30.261414 (+ 66us) server_negotiation.cc:407] Connection header received
0928 14:41:30.261566 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:30.261570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:30.261619 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:30.261711 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:30.262140 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.262622 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.263381 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.263605 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.263718 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:30.266084 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:44131: BlockingRecv error: recv got EOF from 127.1.201.3:44131 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 14:41:30 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:30 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:30.938323 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:30.928649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:42523)
0928 14:41:30.928998 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:30.929014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:30.929160 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:30.929580 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:30.929583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:30.929757 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:30.929913 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:30.929919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.930686 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.930690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:30.931270 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:30.931277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.931763 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.931765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:30.931838 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:30.936567 (+ 4729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:30.936578 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:30.938213 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
W20260928 14:41:30.938525 11012 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:30.928740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:42523 (local address 127.1.201.62:33287)
0928 14:41:30.928999 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:30.929005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:30.929101 (+ 96us) server_negotiation.cc:407] Connection header received
0928 14:41:30.929279 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:30.929282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:30.929336 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:30.929436 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:30.930030 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.930576 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.931388 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:30.931618 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:30.931767 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:30.938331 (+ 6564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:42523: BlockingRecv error: recv got EOF from 127.1.201.1:42523 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
Sep 28 14:41:31 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:31 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:31.105499 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:31.100614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:50105)
0928 14:41:31.100876 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:31.100889 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:31.100988 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:31.101213 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:31.101215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:31.101395 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:31.101486 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:31.101491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.102193 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.102196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:31.102724 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:31.102731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.103103 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.103105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:31.103162 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:31.103618 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:31.103629 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:31.105313 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":72,"threads_started":1}
W20260928 14:41:31.105530 11012 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:31.100706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:50105 (local address 127.1.201.62:33287)
0928 14:41:31.100880 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:31.100884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:31.100960 (+ 76us) server_negotiation.cc:407] Connection header received
0928 14:41:31.101066 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:31.101069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:31.101112 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:31.101178 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:31.101578 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.102103 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.102816 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.103015 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.103085 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:31.105438 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:50105: BlockingRecv error: recv got EOF from 127.1.201.2:50105 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 14:41:31 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:31 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:31.272459 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:31.266787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:39181)
0928 14:41:31.267100 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:31.267115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:31.267232 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:31.267520 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:31.267523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:31.267719 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:31.267811 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:31.267818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.268538 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.268542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:31.269108 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:31.269116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.269637 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.269641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:31.269765 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:31.270381 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:31.270397 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:31.272277 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
W20260928 14:41:31.272490 11012 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:31.266924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:39181 (local address 127.1.201.62:33287)
0928 14:41:31.267081 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:31.267085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:31.267188 (+ 103us) server_negotiation.cc:407] Connection header received
0928 14:41:31.267332 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:31.267335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:31.267387 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:31.267487 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:31.267930 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.268430 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.269234 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.269511 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.269611 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:31.272391 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:39181: BlockingRecv error: recv got EOF from 127.1.201.3:39181 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 14:41:31 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:31 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:31.944577 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:31.939031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:38547)
0928 14:41:31.939331 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:31.939346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:31.939490 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:31.939766 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:31.939769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:31.939983 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:31.940071 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:31.940076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.940916 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.940922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:31.941517 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:31.941525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.942092 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.942096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:31.942185 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:31.942616 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:31.942626 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:31.944418 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
W20260928 14:41:31.944684 11016 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:31.939085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:38547 (local address 127.1.201.62:33287)
0928 14:41:31.939345 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:31.939350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:31.939419 (+ 69us) server_negotiation.cc:407] Connection header received
0928 14:41:31.939599 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:31.939602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:31.939654 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:31.939750 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:31.940187 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.940803 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.941635 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:31.941978 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:31.942087 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:31.944543 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:38547: BlockingRecv error: recv got EOF from 127.1.201.1:38547 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
Sep 28 14:41:32 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:32 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:32.111784 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:32.106484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:45493)
0928 14:41:32.106753 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:32.106765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:32.106862 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:32.107077 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:32.107079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:32.107241 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:32.107330 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:32.107335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.108005 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.108007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:32.108534 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:32.108541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.109016 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.109018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:32.109096 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:32.109685 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:32.109701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:32.111665 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
W20260928 14:41:32.111940 11016 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:32.106540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:45493 (local address 127.1.201.62:33287)
0928 14:41:32.106668 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:32.106671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:32.106830 (+ 159us) server_negotiation.cc:407] Connection header received
0928 14:41:32.106934 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:32.106936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:32.106976 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:32.107043 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:32.107420 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.107919 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.108663 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.108912 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.108989 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:32.111800 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:45493: BlockingRecv error: recv got EOF from 127.1.201.2:45493 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 14:41:32 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:32 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:32.278447 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:32.273179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:47695)
0928 14:41:32.273500 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:32.273519 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:32.273643 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:32.273920 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:32.273923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:32.274110 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:32.274197 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:32.274202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.274946 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.274950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:32.275461 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:32.275468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.275941 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.275945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:32.276029 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:32.276492 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:32.276502 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:32.278290 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":94,"threads_started":1}
W20260928 14:41:32.278501 11016 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:32.273299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:47695 (local address 127.1.201.62:33287)
0928 14:41:32.273460 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:32.273465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:32.273588 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:41:32.273749 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:32.273752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:32.273818 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:32.273898 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:32.274308 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.274830 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.275572 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.275828 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.275937 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:32.278398 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:47695: BlockingRecv error: recv got EOF from 127.1.201.3:47695 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 14:41:32 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:32 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:32.950392 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:32.945284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:43779)
0928 14:41:32.945524 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:32.945536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:32.945616 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:32.945897 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:32.945901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:32.946147 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:32.946244 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:32.946250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.946944 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.946947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:32.947469 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:32.947476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.947869 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.947872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:32.947932 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:32.948475 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:32.948488 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:32.950283 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":78,"threads_started":1}
W20260928 14:41:32.950611 11020 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:32.945341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:43779 (local address 127.1.201.62:33287)
0928 14:41:32.945569 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:32.945572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:32.945584 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:32.945698 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:32.945700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:32.945740 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:32.945876 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:32.946348 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.946859 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.947562 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:32.947779 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:32.947850 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:32.950404 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:43779: BlockingRecv error: recv got EOF from 127.1.201.1:43779 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":56,"threads_started":1}
Sep 28 14:41:33 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:33 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:33.117245 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:33.112513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:34519)
0928 14:41:33.112777 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:33.112788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:33.112868 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:33.113073 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:33.113075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:33.113238 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:33.113323 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:33.113328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.114005 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.114007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:33.114513 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:33.114521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.114869 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.114871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:33.114926 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:33.115476 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:33.115488 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:33.117143 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
W20260928 14:41:33.117408 11020 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:33.112621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:34519 (local address 127.1.201.62:33287)
0928 14:41:33.112774 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:33.112778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:33.112847 (+ 69us) server_negotiation.cc:407] Connection header received
0928 14:41:33.112939 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:33.112940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:33.112980 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:33.113042 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:33.113409 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.113920 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.114598 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.114787 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.114852 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:33.117265 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:34519: BlockingRecv error: recv got EOF from 127.1.201.2:34519 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 14:41:33 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:33 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:33.284475 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:33.279234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:38329)
0928 14:41:33.279492 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:33.279510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:33.279654 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:33.279926 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:33.279929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:33.280133 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:33.280262 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:33.280270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.281088 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.281092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:33.281655 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:33.281669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.282130 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.282134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:33.282221 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:33.282651 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:33.282667 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:33.284316 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
W20260928 14:41:33.284528 11020 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:33.279294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:38329 (local address 127.1.201.62:33287)
0928 14:41:33.279452 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:33.279456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:33.279639 (+ 183us) server_negotiation.cc:407] Connection header received
0928 14:41:33.279767 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:33.279773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:33.279830 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:33.279903 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:33.280359 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.280979 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.281794 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.282019 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.282128 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:33.284425 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:38329: BlockingRecv error: recv got EOF from 127.1.201.3:38329 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 14:41:33 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:33 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:33.955986 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:33.951215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:49075)
0928 14:41:33.951475 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:33.951488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:33.951582 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:33.951802 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:33.951804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:33.951969 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:33.952059 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:33.952064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.952729 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.952732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:33.953278 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:33.953284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.953657 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.953659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:33.953716 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:33.954219 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:33.954229 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:33.955884 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
W20260928 14:41:33.956135 11024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:33.951304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:49075 (local address 127.1.201.62:33287)
0928 14:41:33.951540 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:33.951543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:33.951555 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:33.951658 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:33.951660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:33.951700 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:33.951763 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:33.952152 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.952638 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.953369 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:33.953568 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:33.953634 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:33.956014 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:49075: BlockingRecv error: recv got EOF from 127.1.201.1:49075 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
Sep 28 14:41:34 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:34 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:34.123188 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:34.118036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:53623)
0928 14:41:34.118333 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:34.118353 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:34.118472 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:34.118776 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:34.118779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:34.118979 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:34.119073 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:34.119078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.119804 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.119806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:34.120349 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:34.120356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.120748 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.120751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:34.120822 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:34.121291 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:34.121301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:34.123084 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
W20260928 14:41:34.123313 11024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:34.118063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:53623 (local address 127.1.201.62:33287)
0928 14:41:34.118213 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:34.118217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:34.118426 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:41:34.118579 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:34.118581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:34.118636 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:34.118749 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:34.119196 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.119714 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.120445 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.120655 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.120729 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:34.123209 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:53623: BlockingRecv error: recv got EOF from 127.1.201.2:53623 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 14:41:34 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:34 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:34.290437 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:34.285377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:57159)
0928 14:41:34.285650 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:34.285662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:34.285762 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:34.285995 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:34.285997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:34.286168 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:34.286254 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:34.286259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.286891 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.286893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:34.287434 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:34.287440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.287802 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.287804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:34.287898 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:34.288463 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:34.288476 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:34.290330 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":113,"threads_started":1}
W20260928 14:41:34.290576 11024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:34.285436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:57159 (local address 127.1.201.62:33287)
0928 14:41:34.285575 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:34.285580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:34.285739 (+ 159us) server_negotiation.cc:407] Connection header received
0928 14:41:34.285851 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:34.285853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:34.285897 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:34.285960 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:34.286343 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.286811 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.287518 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.287706 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.287770 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:34.290461 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:57159: BlockingRecv error: recv got EOF from 127.1.201.3:57159 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 14:41:34 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:34 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:34.962370 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:34.956812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:40203)
0928 14:41:34.957108 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:34.957123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:34.957229 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:34.957565 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:34.957569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:34.957868 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:34.957992 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:34.958016 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.958767 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.958769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:34.959344 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:34.959364 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.959856 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.959858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:34.959922 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:34.960486 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:34.960498 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:34.962263 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
W20260928 14:41:34.962544 11028 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:34.956911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:40203 (local address 127.1.201.62:33287)
0928 14:41:34.957176 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:34.957179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:34.957191 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:34.957336 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:34.957339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:34.957420 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:34.957516 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:34.958124 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.958665 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.959475 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:34.959764 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:34.959840 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:34.962383 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:40203: BlockingRecv error: recv got EOF from 127.1.201.1:40203 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
Sep 28 14:41:35 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:35 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:35.128722 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:35.123988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:38459)
0928 14:41:35.124310 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:35.124323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:35.124460 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:35.124685 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:35.124688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:35.124872 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:35.124962 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:35.124968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.125619 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.125621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:35.126145 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:35.126152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.126509 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.126512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:35.126578 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:35.127047 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:35.127057 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:35.128596 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
W20260928 14:41:35.128852 11028 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:35.124030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:38459 (local address 127.1.201.62:33287)
0928 14:41:35.124191 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:35.124195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:35.124403 (+ 208us) server_negotiation.cc:407] Connection header received
0928 14:41:35.124523 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:35.124525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:35.124565 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:35.124626 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:35.125057 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.125536 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.126233 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.126427 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.126493 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:35.128738 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:38459: BlockingRecv error: recv got EOF from 127.1.201.2:38459 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 14:41:35 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:35 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:35.296461 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:35.291260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:46937)
0928 14:41:35.291607 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:35.291618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:35.291753 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:35.292011 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:35.292014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:35.292221 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:35.292308 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:35.292314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.293042 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.293047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:35.293591 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:35.293598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.294100 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.294103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:35.294193 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:35.294654 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:35.294665 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:35.296356 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":114,"threads_started":1}
W20260928 14:41:35.296630 11028 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:35.291313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:46937 (local address 127.1.201.62:33287)
0928 14:41:35.291484 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:35.291489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:35.291698 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:41:35.291850 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:35.291854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:35.291906 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:35.291989 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:35.292423 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.292935 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.293702 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.293991 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.294098 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:35.296480 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:46937: BlockingRecv error: recv got EOF from 127.1.201.3:46937 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 14:41:35 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:35 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:35.974664 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:35.963129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:35877)
0928 14:41:35.963391 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:35.963403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:35.963482 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:35.963704 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:35.963706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:35.963884 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:35.963978 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:35.963983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.964672 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.964673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:35.965174 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:35.965180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.965537 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.965539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:35.965594 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:35.972702 (+ 7108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:35.972713 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:35.974497 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
W20260928 14:41:35.974776 11032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:35.963259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:35877 (local address 127.1.201.62:33287)
0928 14:41:35.963467 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:35.963470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:35.963481 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:41:35.963559 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:35.963561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:35.963600 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:35.963672 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:35.964082 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.964592 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.965261 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:35.965456 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:35.965521 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:35.974632 (+ 9111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:35877: BlockingRecv error: recv got EOF from 127.1.201.1:35877 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
Sep 28 14:41:36 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:36 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:36.134684 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:36.129590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:53541)
0928 14:41:36.129887 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:36.129899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:36.129991 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:36.130188 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:36.130189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:36.130354 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:36.130444 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:36.130448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.131270 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.131272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:36.131818 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:36.131825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.132213 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.132215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:36.132272 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:36.132715 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:36.132726 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:36.134526 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
W20260928 14:41:36.134721 11032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:36.129632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:53541 (local address 127.1.201.62:33287)
0928 14:41:36.129768 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:36.129771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:36.129967 (+ 196us) server_negotiation.cc:407] Connection header received
0928 14:41:36.130067 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:36.130070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:36.130113 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:36.130188 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:36.130533 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.131183 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.131910 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.132106 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.132205 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:36.134632 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:53541: BlockingRecv error: recv got EOF from 127.1.201.2:53541 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 14:41:36 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:36 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:36.302292 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:36.297112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:56201)
0928 14:41:36.297437 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:36.297453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:36.297583 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:36.297874 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:36.297877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:36.298078 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:36.298166 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:36.298171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.298936 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.298941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:36.299484 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:36.299491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.299965 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.299969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:36.300054 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:36.300501 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:36.300512 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:36.302120 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":126,"threads_started":1}
W20260928 14:41:36.302376 11032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:36.297161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:56201 (local address 127.1.201.62:33287)
0928 14:41:36.297340 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:36.297345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:36.297529 (+ 184us) server_negotiation.cc:407] Connection header received
0928 14:41:36.297686 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:36.297689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:36.297743 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:36.297855 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:36.298284 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.298831 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.299609 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.299854 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.299962 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:36.302264 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:56201: BlockingRecv error: recv got EOF from 127.1.201.3:56201 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 14:41:36 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:36 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:36.980273 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:36.975497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:49109)
0928 14:41:36.975748 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:36.975760 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:36.975845 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:36.976055 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:36.976057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:36.976223 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:36.976313 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:36.976318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.976966 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.976968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:36.977482 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:36.977488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.977973 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.977976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:36.978047 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:36.978447 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:36.978457 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:36.980108 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
W20260928 14:41:36.980365 11036 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:36.975559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:49109 (local address 127.1.201.62:33287)
0928 14:41:36.975821 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:36.975824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:36.975836 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:36.975920 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:36.975921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:36.975963 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:36.976024 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:36.976406 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.976884 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.977597 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:36.977870 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:36.977971 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:36.980233 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:49109: BlockingRecv error: recv got EOF from 127.1.201.1:49109 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
Sep 28 14:41:37 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:37 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:37.140924 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:37.135503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:44581)
0928 14:41:37.135794 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:37.135811 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:37.135953 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:37.136229 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:37.136232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:37.136435 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:37.136523 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:37.136528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.137293 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.137299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:37.137847 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:37.137855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.138268 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.138271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:37.138350 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:37.138817 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:37.138828 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:37.140755 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":97,"threads_started":1}
W20260928 14:41:37.140982 11036 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:37.135569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:44581 (local address 127.1.201.62:33287)
0928 14:41:37.135765 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:37.135770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:37.135893 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:41:37.136063 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:37.136065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:37.136110 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:37.136192 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:37.136661 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.137191 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.137958 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.138165 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.138266 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:37.140875 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:44581: BlockingRecv error: recv got EOF from 127.1.201.2:44581 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 14:41:37 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:37 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:37.308385 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:37.303144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:37975)
0928 14:41:37.303457 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:37.303477 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:37.303598 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:37.303862 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:37.303865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:37.304061 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:37.304148 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:37.304154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.304876 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.304881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:37.305422 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:37.305429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.305920 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.305923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:37.306003 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:37.306530 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:37.306540 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:37.308228 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
W20260928 14:41:37.308482 11036 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:37.303255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:37975 (local address 127.1.201.62:33287)
0928 14:41:37.303411 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:37.303416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:37.303548 (+ 132us) server_negotiation.cc:407] Connection header received
0928 14:41:37.303705 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:37.303707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:37.303758 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:37.303840 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:37.304274 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.304771 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.305543 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.305769 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.305919 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:37.308376 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:37975: BlockingRecv error: recv got EOF from 127.1.201.3:37975 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 14:41:37 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:37 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:37.986600 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:37.981123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:34445)
0928 14:41:37.981447 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:37.981461 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:37.981548 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:37.981833 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:37.981836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:37.982079 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:37.982213 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:37.982220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.983133 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.983137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:37.983684 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:37.983691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.984142 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.984145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:37.984226 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:37.984697 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:37.984708 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:37.986437 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":93,"threads_started":1}
W20260928 14:41:37.986698 11040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:37.981196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:34445 (local address 127.1.201.62:33287)
0928 14:41:37.981509 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:37.981514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:37.981529 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:41:37.981623 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:37.981627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:37.981682 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:37.981756 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:37.982342 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.983026 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.983803 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:37.984030 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:37.984138 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:37.986558 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:34445: BlockingRecv error: recv got EOF from 127.1.201.1:34445 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":123,"threads_started":1}
Sep 28 14:41:38 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:38 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:38.148020 11041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:38.141662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:36113)
0928 14:41:38.141917 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:38.141935 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:38.142045 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:38.142364 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:38.142367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:38.142601 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:38.142795 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:38.142805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.143688 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.143691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:38.144303 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:38.144312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.144709 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.144711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:38.144777 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:38.145287 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:38.145305 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:38.147848 (+ 2543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
W20260928 14:41:38.150215 11040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:38.141754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:36113 (local address 127.1.201.62:33287)
0928 14:41:38.141940 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:38.141945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:38.142010 (+ 65us) server_negotiation.cc:407] Connection header received
0928 14:41:38.142168 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:38.142171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:38.142226 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:38.142325 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:38.142902 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.143581 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.144405 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.144607 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.144688 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:38.150107 (+ 5419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:36113: BlockingRecv error: recv got EOF from 127.1.201.2:36113 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 14:41:38 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:38 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:38.314548 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:38.309199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:46463)
0928 14:41:38.309501 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:38.309517 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:38.309621 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:38.309946 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:38.309949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:38.310184 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:38.310273 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:38.310278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.311021 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.311027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:38.311563 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:38.311570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.312023 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.312026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:38.312109 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:38.312626 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:38.312646 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:38.314379 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":112,"threads_started":1}
W20260928 14:41:38.314584 11040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:38.309354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:46463 (local address 127.1.201.62:33287)
0928 14:41:38.309501 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:38.309507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:38.309574 (+ 67us) server_negotiation.cc:407] Connection header received
0928 14:41:38.309747 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:38.309751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:38.309819 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:38.309911 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:38.310390 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.310914 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.311680 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.311892 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.312012 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:38.314490 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:46463: BlockingRecv error: recv got EOF from 127.1.201.3:46463 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 14:41:38 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:38 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:38.992336 11043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:38.987401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:39729)
0928 14:41:38.987660 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:38.987678 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:38.987782 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:38.988013 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:38.988015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:38.988185 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:38.988278 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:38.988283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.988987 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.988989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:38.989486 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:38.989493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.989959 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.989962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:38.990026 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:38.990501 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:38.990512 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:38.992226 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
W20260928 14:41:38.992484 11044 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:38.987449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:39729 (local address 127.1.201.62:33287)
0928 14:41:38.987679 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:38.987682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:38.987733 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:41:38.987860 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:38.987862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:38.987905 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:38.987978 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:38.988375 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.988902 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.989572 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:38.989851 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:38.989944 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:38.992355 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:39729: BlockingRecv error: recv got EOF from 127.1.201.1:39729 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
Sep 28 14:41:39 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:39 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:39.157860 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:39.152778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:59159)
0928 14:41:39.153099 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:39.153113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:39.153232 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:39.153481 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:39.153484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:39.153690 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:39.153830 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:39.153843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.154575 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.154578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:39.155132 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:39.155140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.155494 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.155496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:39.155549 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:39.156016 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:39.156026 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:39.157689 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
W20260928 14:41:39.157974 11044 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:39.152839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:59159 (local address 127.1.201.62:33287)
0928 14:41:39.152992 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:39.152996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:39.153195 (+ 199us) server_negotiation.cc:407] Connection header received
0928 14:41:39.153329 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:39.153336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:39.153378 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:39.153457 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:39.153952 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.154487 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.155225 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.155414 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.155478 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:39.157873 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:59159: BlockingRecv error: recv got EOF from 127.1.201.2:59159 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 14:41:39 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:39 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:39.320674 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:39.315385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:50355)
0928 14:41:39.315826 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:39.315841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:39.315960 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:39.316195 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:39.316197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:39.316359 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:39.316447 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:39.316452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.317158 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.317160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:39.317692 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:39.317699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.318191 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.318193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:39.318255 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:39.318722 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:39.318733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:39.320534 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":88,"threads_started":1}
W20260928 14:41:39.320807 11044 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:39.315771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:50355 (local address 127.1.201.62:33287)
0928 14:41:39.315902 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:39.315913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:39.315927 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:41:39.316043 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:39.316045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:39.316090 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:39.316152 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:39.316539 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.317069 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.317856 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.318086 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.318168 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:39.320662 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:50355: BlockingRecv error: recv got EOF from 127.1.201.3:50355 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 14:41:39 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:39 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:40.000037 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:39.993158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:60941)
0928 14:41:39.993417 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:39.993431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:39.993554 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:39.993868 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:39.993872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:39.994070 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:39.994163 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:39.994170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.995367 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.995371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:39.996344 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:39.996359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.997216 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.997218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:39.997330 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:39.997995 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:39.998006 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:39.999918 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
W20260928 14:41:40.000195 11048 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:39.993245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:60941 (local address 127.1.201.62:33287)
0928 14:41:39.993535 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:39.993541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:39.993559 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:41:39.993667 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:39.993670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:39.993720 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:39.993844 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:39.994277 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.995226 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.996640 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:39.996984 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:39.997077 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:40.000041 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:60941: BlockingRecv error: recv got EOF from 127.1.201.1:60941 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
Sep 28 14:41:40 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:40 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:40.163844 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:40.158535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:54445)
0928 14:41:40.158824 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:40.158837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:40.158933 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:40.159144 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:40.159146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:40.159323 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:40.159428 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:40.159434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:40.160277 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:40.160282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:40.160847 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:40.160855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:40.161242 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:40.161245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:40.161339 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:40.161884 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:40.161894 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:40.163673 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
W20260928 14:41:40.163893 11048 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:40.158627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:54445 (local address 127.1.201.62:33287)
0928 14:41:40.158760 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:40.158763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:40.158904 (+ 141us) server_negotiation.cc:407] Connection header received
0928 14:41:40.159004 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:40.159006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:40.159046 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:40.159112 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:40.159519 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:40.160142 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:40.160931 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:40.161148 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:40.161226 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:40.163786 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:54445: BlockingRecv error: recv got EOF from 127.1.201.2:54445 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 14:41:40 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:40 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:40.327042 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:40.321406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:52361)
0928 14:41:40.321754 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:40.321774 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:40.321932 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:40.322204 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:40.322206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:40.322410 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:40.322518 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:40.322524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:40.323300 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:40.323305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:40.323982 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:40.323990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:40.324436 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:40.324440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:40.324519 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:40.325050 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:40.325070 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:40.326862 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":81,"threads_started":1}
W20260928 14:41:40.327171 11048 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:40.321570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:52361 (local address 127.1.201.62:33287)
0928 14:41:40.321713 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:40.321717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:40.321877 (+ 160us) server_negotiation.cc:407] Connection header received
0928 14:41:40.322050 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:40.322052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:40.322096 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:40.322187 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:40.322629 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:40.323189 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:40.324100 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:40.324325 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:40.324448 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:40.327046 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:52361: BlockingRecv error: recv got EOF from 127.1.201.3:52361 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 14:41:41 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:41 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:41.014835 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:41.000844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:44813)
0928 14:41:41.001124 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:41.001137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:41.001271 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:41.001607 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:41.001610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:41.001838 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:41.001933 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:41.001939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.002621 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.002623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:41.003133 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:41.003140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.003511 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.003513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:41.003571 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:41.012758 (+ 9187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:41.012776 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:41.014716 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
W20260928 14:41:41.014961 11052 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:41.000870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:44813 (local address 127.1.201.62:33287)
0928 14:41:41.001145 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:41.001151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:41.001211 (+ 60us) server_negotiation.cc:407] Connection header received
0928 14:41:41.001392 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:41.001395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:41.001453 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:41.001551 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:41.002033 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.002539 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.003232 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.003428 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.003493 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:41.014839 (+ 11346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:44813: BlockingRecv error: recv got EOF from 127.1.201.1:44813 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
Sep 28 14:41:41 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:41 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:41.169255 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:41.164571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:58721)
0928 14:41:41.164849 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:41.164862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:41.164951 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:41.165137 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:41.165139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:41.165300 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:41.165388 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:41.165392 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.166099 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.166102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:41.166607 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:41.166613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.166975 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.166977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:41.167031 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:41.167437 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:41.167446 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:41.169123 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":118,"threads_started":1}
W20260928 14:41:41.169386 11052 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:41.164686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:58721 (local address 127.1.201.62:33287)
0928 14:41:41.164828 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:41.164831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:41.164910 (+ 79us) server_negotiation.cc:407] Connection header received
0928 14:41:41.165019 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:41.165021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:41.165059 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:41.165128 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:41.165475 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.166009 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.166713 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.166895 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.166958 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:41.169251 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:58721: BlockingRecv error: recv got EOF from 127.1.201.2:58721 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 14:41:41 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:41 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:41.332924 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:41.327824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:60049)
0928 14:41:41.328074 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:41.328085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:41.328180 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:41.328384 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:41.328386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:41.328550 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:41.328638 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:41.328643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.329299 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.329301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:41.329862 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:41.329869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.330319 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.330322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:41.330404 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:41.331008 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:41.331019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:41.332811 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
W20260928 14:41:41.333047 11052 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:41.327879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:60049 (local address 127.1.201.62:33287)
0928 14:41:41.328013 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:41.328016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:41.328154 (+ 138us) server_negotiation.cc:407] Connection header received
0928 14:41:41.328241 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:41.328243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:41.328286 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:41.328348 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:41.328737 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.329217 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.329971 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:41.330185 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:41.330321 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:41.332939 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:60049: BlockingRecv error: recv got EOF from 127.1.201.3:60049 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 14:41:42 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:42 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:42.020514 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:42.015572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:50587)
0928 14:41:42.015896 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:42.015912 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:42.016007 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:42.016237 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:42.016239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:42.016408 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:42.016500 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:42.016505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.017202 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.017204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:42.017701 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:42.017706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.018130 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.018132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:42.018197 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:42.018625 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:42.018635 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:42.020403 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":88,"threads_started":1}
W20260928 14:41:42.020668 11056 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:42.015687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:50587 (local address 127.1.201.62:33287)
0928 14:41:42.015994 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:42.015998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:42.016009 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:41:42.016072 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:42.016074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:42.016130 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:42.016202 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:42.016598 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.017115 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.017804 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.018039 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.018122 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:42.020534 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:50587: BlockingRecv error: recv got EOF from 127.1.201.1:50587 (error 108)
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":67,"threads_started":1}
Sep 28 14:41:42 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:42 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:42.175166 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:42.169979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:45805)
0928 14:41:42.170236 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:42.170249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:42.170345 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:42.170582 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:42.170584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:42.170797 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:42.170890 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:42.170895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.171626 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.171628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:42.172187 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:42.172194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.172643 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.172645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:42.172702 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:42.173277 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:42.173291 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:42.175050 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
W20260928 14:41:42.175318 11056 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:42.170100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:45805 (local address 127.1.201.62:33287)
0928 14:41:42.170249 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:42.170252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:42.170297 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:41:42.170442 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:42.170444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:42.170486 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:42.170545 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:42.170991 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.171526 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.172284 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.172529 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.172639 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:42.175178 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:45805: BlockingRecv error: recv got EOF from 127.1.201.2:45805 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 14:41:42 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:42 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.3@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:42.338891 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:42.333759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.3:35857)
0928 14:41:42.334019 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:42.334031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:42.334155 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:42.334491 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:42.334496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:42.334695 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:42.334813 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:42.334819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.335544 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.335547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:42.336161 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:42.336170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.336564 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.336567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:42.336647 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:42.337100 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:42.337110 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:42.338777 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
W20260928 14:41:42.339048 11056 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:42.333841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:35857 (local address 127.1.201.62:33287)
0928 14:41:42.333966 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:42.333972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:42.334105 (+ 133us) server_negotiation.cc:407] Connection header received
0928 14:41:42.334273 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:42.334277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:42.334349 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:42.334436 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:42.334921 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.335425 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.336267 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:42.336472 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:42.336543 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:42.338925 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.3:35857: BlockingRecv error: recv got EOF from 127.1.201.3:35857 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 14:41:43 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:43 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.1@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:43.026603 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:43.021418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.1:57259)
0928 14:41:43.021702 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:43.021716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:43.021859 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:43.022089 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:43.022091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:43.022273 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:43.022364 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:43.022369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.023128 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.023131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:43.023800 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:43.023809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.024199 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.024201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:43.024258 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:43.024786 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:43.024797 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:43.026446 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
W20260928 14:41:43.026667 11060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:43.021486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:57259 (local address 127.1.201.62:33287)
0928 14:41:43.021728 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:43.021732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:43.021790 (+ 58us) server_negotiation.cc:407] Connection header received
0928 14:41:43.021941 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:43.021943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:43.021987 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:43.022059 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:43.022458 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.023039 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.023893 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.024108 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.024189 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:43.026554 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.1:57259: BlockingRecv error: recv got EOF from 127.1.201.1:57259 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":58,"threads_started":1}
Sep 28 14:41:43 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:41:43 dist-test-slave-0n27 krb5kdc[10001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.201.2@KRBTEST.COM for oryx/127.1.201.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:41:43.181457 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:43.176032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:33287 (local address 127.1.201.2:34807)
0928 14:41:43.176368 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:43.176384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:43.176525 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:43.176818 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:43.176821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:43.177014 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:43.177098 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:43.177103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.177830 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.177834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:43.178345 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:43.178352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.178871 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.178874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:43.178954 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:43.179568 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:43.179584 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:43.181350 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.201.62:33287: Server oryx/127.1.201.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":99,"threads_started":1}
W20260928 14:41:43.181638 11060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:43.176183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:34807 (local address 127.1.201.62:33287)
0928 14:41:43.176351 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:43.176356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:43.176479 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:41:43.176643 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:43.176646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:43.176700 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:43.176785 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:43.177216 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.177703 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.178481 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.178759 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.178868 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:43.181480 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.201.2:34807: BlockingRecv error: recv got EOF from 127.1.201.2:34807 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 14:41:43.246578 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 10584
I20260928 14:41:43.252075 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 10736
I20260928 14:41:43.257130 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 10868
I20260928 14:41:43.262068 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 10514
2026-09-28T14:41:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17784 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T14:41:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:43Z Disabled control of system clock
I20260928 14:41:43.290280 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:46835
--webserver_interface=127.1.201.62
--webserver_port=0
--builtin_ntp_servers=127.1.201.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:46835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:41:43.391455 11065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:43.391718 11065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:43.391752 11065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:43.395340 11065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:43.395385 11065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:43.395397 11065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:43.395412 11065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:43.395424 11065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:43.399631 11065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:46835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:46835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:43.400583 11065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:43.401917 11065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:43.410323 11071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:43.410364 11065 server_base.cc:1034] running on GCE node
W20260928 14:41:43.410279 11073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:43.410274 11070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:43.411087 11065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:43.412089 11065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:43.413239 11065 hybrid_clock.cc:648] HybridClock initialized: now 1790606503413222 us; error 26 us; skew 500 ppm
I20260928 14:41:43.414880 11065 webserver.cc:460] Webserver started at http://127.1.201.62:41197/ using document root <none> and password file <none>
I20260928 14:41:43.415419 11065 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:43.415472 11065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:43.415692 11065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:43.417508 11065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "fa8435c3d66249ad8a645694dff1eec0"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "48a6b51d914352a58e0679442cb57702"
server_key_iv: "55de5c84062dcc9c5b480da9e8f03f25"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.418025 11065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "fa8435c3d66249ad8a645694dff1eec0"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "48a6b51d914352a58e0679442cb57702"
server_key_iv: "55de5c84062dcc9c5b480da9e8f03f25"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.421375 11065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:41:43.423637 11079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:43.424732 11065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:43.424857 11065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "fa8435c3d66249ad8a645694dff1eec0"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "48a6b51d914352a58e0679442cb57702"
server_key_iv: "55de5c84062dcc9c5b480da9e8f03f25"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.424959 11065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:43.433390 11065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:43.434026 11065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:43.434216 11065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:43.440440 11065 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:46835
I20260928 14:41:43.440449 11131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:46835 every 8 connection(s)
I20260928 14:41:43.441336 11065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:43.444375 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:43.445112 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 11065
I20260928 14:41:43.445231 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:43.445519 1828 external_mini_cluster.cc:1428] Setting key 628c9f37bb69788fa42c536e069f5d28
I20260928 14:41:43.450075 11132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0: Bootstrap starting.
I20260928 14:41:43.452736 11132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:43.453526 11132 log.cc:824] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:43.454612 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:43.446862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.1.201.62:46835)
0928 14:41:43.447433 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:43.447447 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:43.447478 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:41:43.448060 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:43.448074 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:43.448469 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:43.448785 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:43.449173 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.450093 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.451327 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.451782 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.451900 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:43.452718 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:43.452723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:43.452733 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:43.453158 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:43.453240 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:43.453259 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:43.453330 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:43.453486 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:41:43.453743 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260928 14:41:43.455600 11132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0: No bootstrap required, opened a new log
I20260928 14:41:43.458997 11132 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8435c3d66249ad8a645694dff1eec0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 46835 } }
I20260928 14:41:43.459213 11132 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:43.459285 11132 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8435c3d66249ad8a645694dff1eec0, State: Initialized, Role: FOLLOWER
I20260928 14:41:43.459733 11132 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8435c3d66249ad8a645694dff1eec0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 46835 } }
I20260928 14:41:43.459864 11132 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:43.459942 11132 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:43.460029 11132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:43.460870 11132 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8435c3d66249ad8a645694dff1eec0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 46835 } }
I20260928 14:41:43.461194 11132 leader_election.cc:302] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa8435c3d66249ad8a645694dff1eec0; no voters:
I20260928 14:41:43.461479 11132 leader_election.cc:288] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:43.461616 11137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:43.462126 11137 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [term 1 LEADER]: Becoming Leader. State: Replica: fa8435c3d66249ad8a645694dff1eec0, State: Running, Role: LEADER
I20260928 14:41:43.462406 11137 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8435c3d66249ad8a645694dff1eec0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 46835 } }
I20260928 14:41:43.462739 11132 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:43.463922 11139 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa8435c3d66249ad8a645694dff1eec0. Latest consensus state: current_term: 1 leader_uuid: "fa8435c3d66249ad8a645694dff1eec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8435c3d66249ad8a645694dff1eec0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 46835 } } }
I20260928 14:41:43.464087 11139 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:43.463938 11138 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa8435c3d66249ad8a645694dff1eec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8435c3d66249ad8a645694dff1eec0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 46835 } } }
I20260928 14:41:43.464212 11138 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0 [sys.catalog]: This master's current role is: LEADER
W20260928 14:41:43.467305 11153 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:41:43.467389 11153 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:41:43.467453 11148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:43.468245 11148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:43.473235 11148 catalog_manager.cc:1347] Generated new cluster ID: fac9fa9877da4e24a882ba13cc0d151b
I20260928 14:41:43.473333 11148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:43.487960 11148 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:43.488806 11148 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:43.499044 11148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa8435c3d66249ad8a645694dff1eec0: Generated new TSK 0
I20260928 14:41:43.499684 11148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:41:43.508224 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:46835
--builtin_ntp_servers=127.1.201.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 14:41:43.619238 11156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:43.619452 11156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:43.619483 11156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:43.623252 11156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:43.623296 11156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:43.623342 11156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:43.627661 11156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:46835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:43.628578 11156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:43.629817 11156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:43.638774 11164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:43.638780 11162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:43.638926 11156 server_base.cc:1034] running on GCE node
W20260928 14:41:43.638780 11161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:43.639667 11156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:43.640234 11156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:43.641400 11156 hybrid_clock.cc:648] HybridClock initialized: now 1790606503641355 us; error 57 us; skew 500 ppm
I20260928 14:41:43.643103 11156 webserver.cc:460] Webserver started at http://127.1.201.1:37785/ using document root <none> and password file <none>
I20260928 14:41:43.643656 11156 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:43.643714 11156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:43.643959 11156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:43.645932 11156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "156b51eb04bb40b4954205228bddef77"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "9eed7866e577b3a624f4da71936a4fda"
server_key_iv: "0d592d9f8d41d8b4c0e385ddb8b9ea6f"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.646453 11156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "156b51eb04bb40b4954205228bddef77"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "9eed7866e577b3a624f4da71936a4fda"
server_key_iv: "0d592d9f8d41d8b4c0e385ddb8b9ea6f"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.649935 11156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:41:43.652211 11170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:43.653306 11156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:43.653468 11156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "156b51eb04bb40b4954205228bddef77"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "9eed7866e577b3a624f4da71936a4fda"
server_key_iv: "0d592d9f8d41d8b4c0e385ddb8b9ea6f"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.653580 11156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:43.668243 11156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:43.668836 11156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:43.669013 11156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:43.669529 11156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:43.670392 11156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:43.670464 11156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:43.670579 11156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:43.670637 11156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:43.679102 11156 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:44057
I20260928 14:41:43.679133 11283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:44057 every 8 connection(s)
I20260928 14:41:43.679935 11156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:43.684880 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 11156
I20260928 14:41:43.685006 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:43.685287 1828 external_mini_cluster.cc:1428] Setting key b4c7524ccf5d998c0edef05bb94065f0
I20260928 14:41:43.687093 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:46835
--builtin_ntp_servers=127.1.201.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:41:43.688756 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:43.681710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:43391 (local address 127.1.201.62:46835)
0928 14:41:43.681903 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:43.681907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:43.682814 (+ 907us) server_negotiation.cc:407] Connection header received
0928 14:41:43.684035 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:43.684037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:43.684085 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:43.684157 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:43.684961 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.685674 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.686902 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.687117 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.687222 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:43.688256 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:43.688259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:43.688261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:43.688279 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:43.688297 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:43.688304 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:43.688392 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:43.688552 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 14:41:43.688618 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:41:43.689504 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:43.682091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:46835 (local address 127.1.201.1:43391)
0928 14:41:43.682684 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:43.682720 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:43.683866 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:43.684312 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:43.684318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:43.684391 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:43.684822 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:43.684833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.685832 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.685836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:43.686782 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:43.686792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.687256 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.687260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:43.687415 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:43.687994 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:43.687996 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:43.688033 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:43.688090 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:43.688406 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:43.688417 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:43.688429 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 14:41:43.688549 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 14:41:43.688802 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
I20260928 14:41:43.690637 11284 heartbeater.cc:347] Connected to a master server at 127.1.201.62:46835
I20260928 14:41:43.690862 11284 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:43.691344 11284 heartbeater.cc:511] Master 127.1.201.62:46835 requested a full tablet report, sending...
I20260928 14:41:43.692903 11096 ts_manager.cc:194] Registered new tserver with Master: 156b51eb04bb40b4954205228bddef77 (127.1.201.1:44057)
I20260928 14:41:43.694581 11096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.1:43391
W20260928 14:41:43.792460 11287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:43.792665 11287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:43.792694 11287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:43.796301 11287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:43.796344 11287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:43.796391 11287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:43.800743 11287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:46835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:43.801610 11287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:43.802835 11287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:43.811527 11292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:43.811549 11287 server_base.cc:1034] running on GCE node
W20260928 14:41:43.811510 11293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:43.811518 11295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:43.812186 11287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:43.812831 11287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:43.814038 11287 hybrid_clock.cc:648] HybridClock initialized: now 1790606503814025 us; error 25 us; skew 500 ppm
I20260928 14:41:43.815774 11287 webserver.cc:460] Webserver started at http://127.1.201.2:34185/ using document root <none> and password file <none>
I20260928 14:41:43.816322 11287 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:43.816409 11287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:43.816612 11287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:43.818521 11287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "8d3b4d8d9e3649babbb5baf5208b2330"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "c7a1acaa11a59a1a640c5ed2dfa066a5"
server_key_iv: "5db745585d7ce7d1bb509d055489696d"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.819025 11287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "8d3b4d8d9e3649babbb5baf5208b2330"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "c7a1acaa11a59a1a640c5ed2dfa066a5"
server_key_iv: "5db745585d7ce7d1bb509d055489696d"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.822499 11287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:41:43.824846 11301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:43.826012 11287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:43.826148 11287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "8d3b4d8d9e3649babbb5baf5208b2330"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "c7a1acaa11a59a1a640c5ed2dfa066a5"
server_key_iv: "5db745585d7ce7d1bb509d055489696d"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.826257 11287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:43.839151 11287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:43.839803 11287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:43.839991 11287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:43.840588 11287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:43.841540 11287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:43.841612 11287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:43.841688 11287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:43.841740 11287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:43.849992 11287 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:33665
I20260928 14:41:43.850014 11414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:33665 every 8 connection(s)
I20260928 14:41:43.850844 11287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:43.853245 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 11287
I20260928 14:41:43.853358 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:43.853626 1828 external_mini_cluster.cc:1428] Setting key ed8b86803b8fb0304e2674f8f58a4c8f
I20260928 14:41:43.855707 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:46835
--builtin_ntp_servers=127.1.201.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:41:43.860559 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:43.852591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:59141 (local address 127.1.201.62:46835)
0928 14:41:43.852736 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:43.852741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:43.853849 (+ 1108us) server_negotiation.cc:407] Connection header received
0928 14:41:43.855281 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:43.855284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:43.855333 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:43.855413 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:43.856340 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.857165 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.858754 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.858972 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.859077 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:43.860076 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:43.860079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:43.860082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:43.860093 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:43.860120 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:43.860128 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:43.860199 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:43.860346 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 14:41:43.860409 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:41:43.861366 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:43.853041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:46835 (local address 127.1.201.2:59141)
0928 14:41:43.853685 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:43.853734 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:43.855090 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:43.855554 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:43.855559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:43.855647 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:43.856191 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:43.856205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.857303 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.857309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:43.858636 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:43.858646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:43.859114 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:43.859118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:43.859278 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:43.859836 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:43.859838 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:43.859874 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:43.859931 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:43.860212 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:43.860226 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:43.860231 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:41:43.860345 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 14:41:43.860616 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
I20260928 14:41:43.862468 11415 heartbeater.cc:347] Connected to a master server at 127.1.201.62:46835
I20260928 14:41:43.862712 11415 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:43.863195 11415 heartbeater.cc:511] Master 127.1.201.62:46835 requested a full tablet report, sending...
I20260928 14:41:43.864274 11096 ts_manager.cc:194] Registered new tserver with Master: 8d3b4d8d9e3649babbb5baf5208b2330 (127.1.201.2:33665)
I20260928 14:41:43.865090 11096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.2:59141
W20260928 14:41:43.958707 11418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:43.958931 11418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:43.958966 11418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:43.962801 11418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:43.962865 11418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:43.962915 11418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:43.967315 11418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:46835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:43.968145 11418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:43.969270 11418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:43.977772 11423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:43.977798 11426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:43.977865 11418 server_base.cc:1034] running on GCE node
W20260928 14:41:43.977818 11424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:43.978379 11418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:43.978900 11418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:43.980070 11418 hybrid_clock.cc:648] HybridClock initialized: now 1790606503980059 us; error 23 us; skew 500 ppm
I20260928 14:41:43.981863 11418 webserver.cc:460] Webserver started at http://127.1.201.3:36855/ using document root <none> and password file <none>
I20260928 14:41:43.982365 11418 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:43.982443 11418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:43.982621 11418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:43.986020 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "882af82d43ca4c9cb9a762c22ec17cee"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "d2b4fe2534a657674a7302c546a162d5"
server_key_iv: "22197e2a31b5ac6665b09f62ead2ba0e"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.986514 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "882af82d43ca4c9cb9a762c22ec17cee"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "d2b4fe2534a657674a7302c546a162d5"
server_key_iv: "22197e2a31b5ac6665b09f62ead2ba0e"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.989734 11418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:41:43.992136 11432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:43.993224 11418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:43.993342 11418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "882af82d43ca4c9cb9a762c22ec17cee"
format_stamp: "Formatted at 2026-09-28 14:41:43 on dist-test-slave-0n27"
server_key: "d2b4fe2534a657674a7302c546a162d5"
server_key_iv: "22197e2a31b5ac6665b09f62ead2ba0e"
server_key_version: "encryptionkey@0"
I20260928 14:41:43.993443 11418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:44.005349 11418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:44.005975 11418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:44.006143 11418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:44.006675 11418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:44.007525 11418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:44.007588 11418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.007654 11418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:44.007704 11418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.015904 11418 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:38365
I20260928 14:41:44.015934 11545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:38365 every 8 connection(s)
I20260928 14:41:44.016736 11418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:41:44.021953 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 11418
I20260928 14:41:44.022064 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:44.022310 1828 external_mini_cluster.cc:1428] Setting key f89ed40f1e8c7d4d605928ef6c8b48ff
I20260928 14:41:44.025334 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:44.018387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:45507 (local address 127.1.201.62:46835)
0928 14:41:44.018558 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:44.018563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:44.019505 (+ 942us) server_negotiation.cc:407] Connection header received
0928 14:41:44.020747 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:44.020749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:44.020795 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:44.020868 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:44.021655 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:44.022281 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:44.023658 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:44.023866 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:44.023966 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:44.024889 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:44.024893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:44.024895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:44.024913 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:44.024929 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:44.024935 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:44.025003 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:44.025151 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 14:41:44.025211 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:41:44.026099 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:44.018779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:46835 (local address 127.1.201.3:45507)
0928 14:41:44.019362 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:44.019399 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:44.020525 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:44.021015 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:44.021021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:44.021096 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:44.021540 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:44.021551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:44.022441 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:44.022444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:44.023551 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:44.023560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:44.024000 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:44.024004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:44.024164 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:44.024657 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:44.024659 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:44.024703 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:44.024758 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:44.025015 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:44.025024 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:44.025029 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:41:44.025140 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:41:44.025377 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":105,"threads_started":1}
I20260928 14:41:44.027038 11546 heartbeater.cc:347] Connected to a master server at 127.1.201.62:46835
I20260928 14:41:44.027235 11546 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:44.027685 11546 heartbeater.cc:511] Master 127.1.201.62:46835 requested a full tablet report, sending...
I20260928 14:41:44.028587 11096 ts_manager.cc:194] Registered new tserver with Master: 882af82d43ca4c9cb9a762c22ec17cee (127.1.201.3:38365)
I20260928 14:41:44.029178 11096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.3:45507
I20260928 14:41:44.035715 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:41:44.038136 11135 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:44.037398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.1.201.62:46835)
0928 14:41:44.037522 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:44.037527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:44.037697 (+ 170us) server_negotiation.cc:407] Connection header received
0928 14:41:44.037754 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:44.037755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:44.037845 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:44.037902 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:44.038057 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47898: BlockingRecv error: recv got EOF from 127.0.0.1:47898 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:41:44.038410 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 11156
I20260928 14:41:44.043594 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 11287
I20260928 14:41:44.048837 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 11418
I20260928 14:41:44.053109 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 11065
2026-09-28T14:41:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (790 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T14:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:44Z Disabled control of system clock
I20260928 14:41:44.080787 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40019
--webserver_interface=127.1.201.62
--webserver_port=0
--builtin_ntp_servers=127.1.201.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:40019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:41:44.181224 11557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:44.181459 11557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:44.181495 11557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:44.185352 11557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:44.185397 11557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:44.185411 11557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:44.185425 11557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:44.185439 11557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:44.189713 11557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:40019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:44.190718 11557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:44.192023 11557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:44.200189 11563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:44.200284 11562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:44.200219 11557 server_base.cc:1034] running on GCE node
W20260928 14:41:44.200165 11565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:44.200953 11557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:44.202102 11557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:44.203281 11557 hybrid_clock.cc:648] HybridClock initialized: now 1790606504203267 us; error 26 us; skew 500 ppm
I20260928 14:41:44.205081 11557 webserver.cc:460] Webserver started at http://127.1.201.62:33433/ using document root <none> and password file <none>
I20260928 14:41:44.205685 11557 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:44.205775 11557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:44.206019 11557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:44.208024 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "dc3faeb2260b430d9a09c05f6d2b500f"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "e985c8f6b83992d8b048fe0a45474234"
server_key_iv: "ff3a505a69d9fd53da1ae7ea70895a75"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.208638 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "dc3faeb2260b430d9a09c05f6d2b500f"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "e985c8f6b83992d8b048fe0a45474234"
server_key_iv: "ff3a505a69d9fd53da1ae7ea70895a75"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.212352 11557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:41:44.214702 11571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.215930 11557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:44.216084 11557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "dc3faeb2260b430d9a09c05f6d2b500f"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "e985c8f6b83992d8b048fe0a45474234"
server_key_iv: "ff3a505a69d9fd53da1ae7ea70895a75"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.216205 11557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:44.220474 11557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:44.221083 11557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:44.221267 11557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:44.227393 11557 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:40019
I20260928 14:41:44.227409 11623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:40019 every 8 connection(s)
I20260928 14:41:44.228199 11557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:44.231024 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:44.235838 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 11557
I20260928 14:41:44.235966 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:44.236220 1828 external_mini_cluster.cc:1428] Setting key c3afe2dc9213b8f29a62d4206f6d681e
I20260928 14:41:44.236351 11624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f: Bootstrap starting.
I20260928 14:41:44.238885 11624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:44.239675 11624 log.cc:824] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:44.241623 11627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:44.237462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.1.201.62:40019)
0928 14:41:44.238128 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:44.238148 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:44.238191 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:41:44.238788 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:44.238803 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:44.239186 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:44.239452 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:44.240253 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:44.240261 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:44.240278 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:44.240293 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:44.240359 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:44.240384 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:44.240491 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:44.240663 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:41:44.240935 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
I20260928 14:41:44.241660 11624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f: No bootstrap required, opened a new log
I20260928 14:41:44.245255 11624 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3faeb2260b430d9a09c05f6d2b500f" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40019 } }
I20260928 14:41:44.245496 11624 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:44.245582 11624 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3faeb2260b430d9a09c05f6d2b500f, State: Initialized, Role: FOLLOWER
I20260928 14:41:44.246035 11624 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3faeb2260b430d9a09c05f6d2b500f" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40019 } }
I20260928 14:41:44.246194 11624 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:44.246272 11624 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:44.246361 11624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:44.247151 11624 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3faeb2260b430d9a09c05f6d2b500f" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40019 } }
I20260928 14:41:44.247483 11624 leader_election.cc:302] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc3faeb2260b430d9a09c05f6d2b500f; no voters:
I20260928 14:41:44.247779 11624 leader_election.cc:288] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:44.247882 11629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:44.248142 11629 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [term 1 LEADER]: Becoming Leader. State: Replica: dc3faeb2260b430d9a09c05f6d2b500f, State: Running, Role: LEADER
I20260928 14:41:44.248462 11629 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3faeb2260b430d9a09c05f6d2b500f" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40019 } }
I20260928 14:41:44.248927 11624 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:44.250140 11630 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc3faeb2260b430d9a09c05f6d2b500f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3faeb2260b430d9a09c05f6d2b500f" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40019 } } }
I20260928 14:41:44.250130 11631 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc3faeb2260b430d9a09c05f6d2b500f. Latest consensus state: current_term: 1 leader_uuid: "dc3faeb2260b430d9a09c05f6d2b500f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3faeb2260b430d9a09c05f6d2b500f" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40019 } } }
I20260928 14:41:44.250272 11630 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:44.250272 11631 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:44.250775 11638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:44.253597 11638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:44.258486 11638 catalog_manager.cc:1347] Generated new cluster ID: 2763fae5db3040859e252a621a981297
I20260928 14:41:44.258544 11638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:44.269021 11638 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:44.269321 11638 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:44.276386 11638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc3faeb2260b430d9a09c05f6d2b500f: Generated new TSK 0
I20260928 14:41:44.277022 11638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:41:44.280925 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40019
--builtin_ntp_servers=127.1.201.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:41:44.384238 11648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:44.384459 11648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:44.384492 11648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:44.388278 11648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:44.388326 11648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:44.388420 11648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:44.392876 11648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:44.393886 11648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:44.395175 11648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:44.403781 11656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:44.403759 11653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:44.403789 11654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:44.404039 11648 server_base.cc:1034] running on GCE node
I20260928 14:41:44.404441 11648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:44.405052 11648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:44.406199 11648 hybrid_clock.cc:648] HybridClock initialized: now 1790606504406168 us; error 44 us; skew 500 ppm
I20260928 14:41:44.408510 11648 webserver.cc:460] Webserver started at http://127.1.201.1:40711/ using document root <none> and password file <none>
I20260928 14:41:44.409301 11648 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:44.409394 11648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:44.409642 11648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:44.412989 11648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "3b98369a2873485e80e1230f62a1708f"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "3f39ae73bbe8fc60d0067073abf65c64"
server_key_iv: "ae1eb8e6395a5c1ac4ba34fc8995a670"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.413846 11648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "3b98369a2873485e80e1230f62a1708f"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "3f39ae73bbe8fc60d0067073abf65c64"
server_key_iv: "ae1eb8e6395a5c1ac4ba34fc8995a670"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.419426 11648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 14:41:44.423298 11662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.424857 11648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:41:44.425066 11648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "3b98369a2873485e80e1230f62a1708f"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "3f39ae73bbe8fc60d0067073abf65c64"
server_key_iv: "ae1eb8e6395a5c1ac4ba34fc8995a670"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.425312 11648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:44.430435 11648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:44.431143 11648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:44.431354 11648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:44.431857 11648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:44.432761 11648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:44.432803 11648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.432837 11648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:44.432850 11648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.441831 11648 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:42205
I20260928 14:41:44.441892 11775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:42205 every 8 connection(s)
I20260928 14:41:44.442679 11648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:44.446774 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 11648
I20260928 14:41:44.446895 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:44.447197 1828 external_mini_cluster.cc:1428] Setting key 1513845991c2d64afa2c5a5981dc764e
I20260928 14:41:44.448895 11627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:44.444328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:47633 (local address 127.1.201.62:40019)
0928 14:41:44.444455 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:44.444459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:44.445412 (+ 953us) server_negotiation.cc:407] Connection header received
0928 14:41:44.446906 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:44.446910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:44.446953 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:44.447048 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:44.448405 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:44.448422 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:44.448424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:44.448436 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:44.448456 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:44.448468 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:44.448537 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:44.448717 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 14:41:44.448759 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:41:44.449226 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40019
--builtin_ntp_servers=127.1.201.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 14:41:44.450042 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:44.444669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40019 (local address 127.1.201.1:47633)
0928 14:41:44.445278 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:44.445313 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:44.446651 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:44.447234 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:44.447245 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:44.447365 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:44.448049 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:44.448052 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:44.448105 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:44.448213 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:44.448539 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:44.448552 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:44.448560 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:41:44.448751 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 14:41:44.449000 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260928 14:41:44.451454 11776 heartbeater.cc:347] Connected to a master server at 127.1.201.62:40019
I20260928 14:41:44.451722 11776 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:44.452221 11776 heartbeater.cc:511] Master 127.1.201.62:40019 requested a full tablet report, sending...
I20260928 14:41:44.454965 11588 ts_manager.cc:194] Registered new tserver with Master: 3b98369a2873485e80e1230f62a1708f (127.1.201.1:42205)
W20260928 14:41:44.571416 11779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:44.571666 11779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:44.571703 11779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:44.575826 11779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:44.575892 11779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:44.575978 11779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:44.584218 11779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:44.585366 11779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:44.587153 11779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:44.596642 11785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:44.596657 11787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:44.596874 11779 server_base.cc:1034] running on GCE node
W20260928 14:41:44.597000 11784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:44.597460 11779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:44.598156 11779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:44.599339 11779 hybrid_clock.cc:648] HybridClock initialized: now 1790606504599300 us; error 48 us; skew 500 ppm
I20260928 14:41:44.601225 11779 webserver.cc:460] Webserver started at http://127.1.201.2:42629/ using document root <none> and password file <none>
I20260928 14:41:44.601743 11779 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:44.601872 11779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:44.602136 11779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:44.604211 11779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "8760bbf8f1d44abca14b225843cf02c4"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "0d03d2938eb115468f098e8debdd3eef"
server_key_iv: "6c8b9f1af7645c191bd79cbe823366e0"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.604794 11779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "8760bbf8f1d44abca14b225843cf02c4"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "0d03d2938eb115468f098e8debdd3eef"
server_key_iv: "6c8b9f1af7645c191bd79cbe823366e0"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.608505 11779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260928 14:41:44.610926 11793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.612142 11779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:41:44.612257 11779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "8760bbf8f1d44abca14b225843cf02c4"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "0d03d2938eb115468f098e8debdd3eef"
server_key_iv: "6c8b9f1af7645c191bd79cbe823366e0"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.612402 11779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:44.616786 11779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:44.617349 11779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:44.617530 11779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:44.618196 11779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:44.619107 11779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:44.619154 11779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.619220 11779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:44.619251 11779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.627017 11779 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:44995
I20260928 14:41:44.627039 11906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:44995 every 8 connection(s)
I20260928 14:41:44.627781 11779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:44.633198 11627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:44.629425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:60337 (local address 127.1.201.62:40019)
0928 14:41:44.629578 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:44.629583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:44.630596 (+ 1013us) server_negotiation.cc:407] Connection header received
0928 14:41:44.631809 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:44.631811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:44.631847 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:44.631928 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:44.632843 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:44.632846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:44.632848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:44.632860 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:44.632879 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:44.632886 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:44.632956 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:44.633059 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 14:41:44.633094 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:41:44.634012 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:44.629884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40019 (local address 127.1.201.2:60337)
0928 14:41:44.630455 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:44.630490 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:44.631614 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:44.632068 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:44.632073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:44.632149 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:44.632616 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:44.632617 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:44.632651 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:44.632718 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:44.632953 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:44.632959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:44.632963 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:41:44.633073 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:41:44.633264 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260928 14:41:44.635008 11907 heartbeater.cc:347] Connected to a master server at 127.1.201.62:40019
I20260928 14:41:44.635212 11907 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:44.635566 11907 heartbeater.cc:511] Master 127.1.201.62:40019 requested a full tablet report, sending...
I20260928 14:41:44.636682 11588 ts_manager.cc:194] Registered new tserver with Master: 8760bbf8f1d44abca14b225843cf02c4 (127.1.201.2:44995)
I20260928 14:41:44.636866 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 11779
I20260928 14:41:44.636986 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:44.637337 1828 external_mini_cluster.cc:1428] Setting key 2729f8b9a49b3f6ca523a4a7c1f714c5
I20260928 14:41:44.639029 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40019
--builtin_ntp_servers=127.1.201.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:41:44.752211 11910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:44.752558 11910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:44.752647 11910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:44.758517 11910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:44.758569 11910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:44.758666 11910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:44.763916 11910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:44.765141 11910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:44.766798 11910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:44.776051 11915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:44.776077 11918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:44.776093 11916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:44.776329 11910 server_base.cc:1034] running on GCE node
I20260928 14:41:44.776750 11910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:44.777292 11910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:44.778434 11910 hybrid_clock.cc:648] HybridClock initialized: now 1790606504778401 us; error 43 us; skew 500 ppm
I20260928 14:41:44.780275 11910 webserver.cc:460] Webserver started at http://127.1.201.3:42419/ using document root <none> and password file <none>
I20260928 14:41:44.780779 11910 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:44.780833 11910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:44.780985 11910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:44.782857 11910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "c8bd4261d028482ba8a85b81f37ada57"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "7562b1fb16f6248a74aadbd286978c6e"
server_key_iv: "a0230b06541d1dc4aa6b8cf669fe8925"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.783313 11910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "c8bd4261d028482ba8a85b81f37ada57"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "7562b1fb16f6248a74aadbd286978c6e"
server_key_iv: "a0230b06541d1dc4aa6b8cf669fe8925"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.786427 11910 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:41:44.788589 11924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.789628 11910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:44.789724 11910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "c8bd4261d028482ba8a85b81f37ada57"
format_stamp: "Formatted at 2026-09-28 14:41:44 on dist-test-slave-0n27"
server_key: "7562b1fb16f6248a74aadbd286978c6e"
server_key_iv: "a0230b06541d1dc4aa6b8cf669fe8925"
server_key_version: "encryptionkey@0"
I20260928 14:41:44.789865 11910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:44.793751 11910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:44.794272 11910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:44.794431 11910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:44.794894 11910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:44.795676 11910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:44.795717 11910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.795776 11910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:44.795809 11910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:44.804381 11910 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:46723
I20260928 14:41:44.804399 12037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:46723 every 8 connection(s)
I20260928 14:41:44.805236 11910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:41:44.810807 11627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:44.806885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:54053 (local address 127.1.201.62:40019)
0928 14:41:44.807014 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:44.807017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:44.807970 (+ 953us) server_negotiation.cc:407] Connection header received
0928 14:41:44.809176 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:44.809179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:44.809239 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:44.809338 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:44.810322 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:44.810325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:44.810328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:41:44.810339 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:44.810361 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:41:44.810376 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:44.810468 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:44.810626 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 14:41:44.810680 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:41:44.811551 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:44.807251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40019 (local address 127.1.201.3:54053)
0928 14:41:44.807839 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:44.807874 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:44.808978 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:44.809500 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:44.809506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:44.809581 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:44.810072 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:41:44.810074 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:44.810108 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:41:44.810167 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:44.810454 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:44.810462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:44.810467 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:41:44.810684 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 14:41:44.810903 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":101,"threads_started":1}
I20260928 14:41:44.812527 12038 heartbeater.cc:347] Connected to a master server at 127.1.201.62:40019
I20260928 14:41:44.812706 12038 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:44.813007 12038 heartbeater.cc:511] Master 127.1.201.62:40019 requested a full tablet report, sending...
I20260928 14:41:44.813890 11588 ts_manager.cc:194] Registered new tserver with Master: c8bd4261d028482ba8a85b81f37ada57 (127.1.201.3:46723)
I20260928 14:41:44.814772 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 11910
I20260928 14:41:44.814893 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:44.815249 1828 external_mini_cluster.cc:1428] Setting key 5f489bd13cdc0ea05e80f1f8acbda644
I20260928 14:41:44.816939 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:41:44.819425 11627 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:41:44.818629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.1.201.62:40019)
0928 14:41:44.818759 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:44.818763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:44.818914 (+ 151us) server_negotiation.cc:407] Connection header received
0928 14:41:44.819035 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:44.819037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:44.819082 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:44.819151 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:44.819346 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34462: BlockingRecv error: recv got EOF from 127.0.0.1:34462 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:41:44.819887 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 11648
I20260928 14:41:44.825696 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 11779
I20260928 14:41:44.830581 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 11910
I20260928 14:41:44.835469 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 11557
2026-09-28T14:41:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (781 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:41:44 dist-test-slave-0n27 krb5kdc[12048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:41:44 dist-test-slave-0n27 krb5kdc[12048](info): set up 2 sockets
Sep 28 14:41:44 dist-test-slave-0n27 krb5kdc[12048](info): commencing operation
krb5kdc: starting...
W20260928 14:41:45.807221 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:41:45 dist-test-slave-0n27 krb5kdc[12048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:41:45.917706 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:37751
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:37751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:46.018965 12061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:46.019212 12061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:46.019275 12061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:46.022960 12061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:41:46.023038 12061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:46.023072 12061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:46.023123 12061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:41:46.023154 12061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:41:46.027974 12061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:37751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:37751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:46.028995 12061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:46.029896 12061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:46.037669 12067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:46.037657 12066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:46.037736 12061 server_base.cc:1034] running on GCE node
W20260928 14:41:46.037657 12069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:46.038384 12061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:46.039304 12061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:46.040452 12061 hybrid_clock.cc:648] HybridClock initialized: now 1790606506040426 us; error 38 us; skew 500 ppm
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606506, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:46.047928 12061 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:41:46.048740 12061 webserver.cc:460] Webserver started at http://127.1.201.62:40847/ using document root <none> and password file <none>
I20260928 14:41:46.049242 12061 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:46.049294 12061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:46.049515 12061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:46.051496 12061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "76d5f7145e304d63a88637e714cfc317"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "da3de6833f248c6e27b7d58cb8462ea8"
server_key_iv: "2008d706ddd3e869a7ba52b382d46cab"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.052071 12061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "76d5f7145e304d63a88637e714cfc317"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "da3de6833f248c6e27b7d58cb8462ea8"
server_key_iv: "2008d706ddd3e869a7ba52b382d46cab"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.055544 12061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:46.057713 12076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:46.058830 12061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:46.058966 12061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "76d5f7145e304d63a88637e714cfc317"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "da3de6833f248c6e27b7d58cb8462ea8"
server_key_iv: "2008d706ddd3e869a7ba52b382d46cab"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.059068 12061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:46.102552 12061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:46.105254 12061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:46.105444 12061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:46.111573 12061 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:37751
I20260928 14:41:46.111631 12128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:37751 every 8 connection(s)
I20260928 14:41:46.112524 12061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:41:46.113471 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 12061
I20260928 14:41:46.113592 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:41:46.113961 1828 external_mini_cluster.cc:1428] Setting key f017cca9150ea6440d9dffa6926c0482
I20260928 14:41:46.115304 12129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:46.120342 12129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317: Bootstrap starting.
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:46.122879 12129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:46.123631 12129 log.cc:824] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:46.125747 12129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317: No bootstrap required, opened a new log
I20260928 14:41:46.127405 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.115198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.1.201.62:37751)
0928 14:41:46.115587 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.115600 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.115648 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:41:46.116217 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.116231 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.116510 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.116833 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:46.117542 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.118517 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.119295 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.119724 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.119878 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.122504 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:46.122510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:46.122526 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:46.122557 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:46.124613 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:46.125092 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:46.125096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:46.125100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:46.125199 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:46.125463 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:46.125465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:46.125466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:46.125939 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:46.126092 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.126414 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.126731 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260928 14:41:46.129376 12129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d5f7145e304d63a88637e714cfc317" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37751 } }
I20260928 14:41:46.129614 12129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:46.129704 12129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d5f7145e304d63a88637e714cfc317, State: Initialized, Role: FOLLOWER
I20260928 14:41:46.130131 12129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d5f7145e304d63a88637e714cfc317" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37751 } }
I20260928 14:41:46.130282 12129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:41:46.130369 12129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:41:46.130484 12129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:46.131414 12129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d5f7145e304d63a88637e714cfc317" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37751 } }
I20260928 14:41:46.131749 12129 leader_election.cc:302] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76d5f7145e304d63a88637e714cfc317; no voters:
I20260928 14:41:46.132056 12129 leader_election.cc:288] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:41:46.132182 12134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:46.132413 12134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [term 1 LEADER]: Becoming Leader. State: Replica: 76d5f7145e304d63a88637e714cfc317, State: Running, Role: LEADER
I20260928 14:41:46.132699 12134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d5f7145e304d63a88637e714cfc317" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37751 } }
I20260928 14:41:46.133280 12129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:41:46.134323 12136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d5f7145e304d63a88637e714cfc317. Latest consensus state: current_term: 1 leader_uuid: "76d5f7145e304d63a88637e714cfc317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d5f7145e304d63a88637e714cfc317" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37751 } } }
I20260928 14:41:46.134449 12136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [sys.catalog]: This master's current role is: LEADER
I20260928 14:41:46.134752 12135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76d5f7145e304d63a88637e714cfc317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d5f7145e304d63a88637e714cfc317" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37751 } } }
I20260928 14:41:46.134869 12135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317 [sys.catalog]: This master's current role is: LEADER
W20260928 14:41:46.138007 12149 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:41:46.138110 12149 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:41:46.138170 12143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:41:46.138899 12143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:41:46.144115 12143 catalog_manager.cc:1347] Generated new cluster ID: 21eeb87bf6504e80830de5628eb5d964
I20260928 14:41:46.144173 12143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:41:46.156670 12143 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:41:46.157866 12143 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:41:46.163239 12143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317: Generated new TSK 0
I20260928 14:41:46.163944 12143 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:41:46.194283 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:37751
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:46.299089 12155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:46.299302 12155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:46.299338 12155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:46.303123 12155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:46.303169 12155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:46.303237 12155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:41:46.308248 12155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:37751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:46.309145 12155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:46.310146 12155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:46.318713 12161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:46.318750 12163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:46.318745 12160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:46.318849 12155 server_base.cc:1034] running on GCE node
I20260928 14:41:46.319325 12155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:46.319898 12155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:46.321050 12155 hybrid_clock.cc:648] HybridClock initialized: now 1790606506321019 us; error 43 us; skew 500 ppm
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606506, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:46.323508 12155 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:41:46.324262 12155 webserver.cc:460] Webserver started at http://127.1.201.1:32779/ using document root <none> and password file <none>
I20260928 14:41:46.324790 12155 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:46.324877 12155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:46.325093 12155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:46.326947 12155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "154158dbd03943bda8c0dae8bd268afa"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "c0dd55563dbba9d7bb3e40ae64471b30"
server_key_iv: "8282dd8594ac68fc20db17d230bf9813"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.327459 12155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "154158dbd03943bda8c0dae8bd268afa"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "c0dd55563dbba9d7bb3e40ae64471b30"
server_key_iv: "8282dd8594ac68fc20db17d230bf9813"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.330849 12155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 14:41:46.333153 12170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:46.334383 12155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:41:46.334501 12155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "154158dbd03943bda8c0dae8bd268afa"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "c0dd55563dbba9d7bb3e40ae64471b30"
server_key_iv: "8282dd8594ac68fc20db17d230bf9813"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.334602 12155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:46.353183 12155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:46.356241 12155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:46.356422 12155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:46.357030 12155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:46.357904 12155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:46.357970 12155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:46.358042 12155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:46.358085 12155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:46.367108 12155 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:38847
I20260928 14:41:46.367127 12283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:38847 every 8 connection(s)
I20260928 14:41:46.367964 12155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:41:46.370167 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 12155
I20260928 14:41:46.370270 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:41:46.370527 1828 external_mini_cluster.cc:1428] Setting key eaf77f7c179183fd91146a844e6d311a
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606506, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
I20260928 14:41:46.382684 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.369894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:37033 (local address 127.1.201.62:37751)
0928 14:41:46.370039 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.370043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.371022 (+ 979us) server_negotiation.cc:407] Connection header received
0928 14:41:46.372693 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.372695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.372740 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.372810 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:46.374305 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.375166 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.376311 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.376722 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.376847 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.379118 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:46.379122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:46.379125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:46.379151 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:46.380582 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:46.381048 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:46.381050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:46.381052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:46.381173 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:46.381525 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:46.381528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:46.381530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:46.381716 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:46.381857 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.382388 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.382517 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:41:46.383700 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.370342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:37751 (local address 127.1.201.1:37033)
0928 14:41:46.370845 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:46.370903 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:46.372472 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:46.373005 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:46.373015 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:46.373527 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:46.374156 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.374172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.375286 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.375288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.376157 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.376167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.376873 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.376876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.377003 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.377548 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:46.377562 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:46.378929 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:46.380685 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:46.380689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:46.380700 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:46.380955 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:46.381269 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:46.381272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:46.381274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:46.381417 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:46.381844 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:46.381851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:46.382148 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 14:41:46.382332 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 14:41:46.382715 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":94,"threads_started":1}
I20260928 14:41:46.384972 12284 heartbeater.cc:347] Connected to a master server at 127.1.201.62:37751
I20260928 14:41:46.385247 12284 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:46.385752 12284 heartbeater.cc:511] Master 127.1.201.62:37751 requested a full tablet report, sending...
I20260928 14:41:46.387182 12093 ts_manager.cc:194] Registered new tserver with Master: 154158dbd03943bda8c0dae8bd268afa (127.1.201.1:38847)
I20260928 14:41:46.388276 12093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:37033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:41:46.398260 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:37751
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:46.498344 12289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:46.498628 12289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:46.498719 12289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:46.502483 12289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:46.502561 12289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:46.502660 12289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:41:46.508605 12289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:37751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:46.509591 12289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:46.510572 12289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:46.519532 12294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:46.519583 12295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:46.519706 12297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:46.519577 12289 server_base.cc:1034] running on GCE node
I20260928 14:41:46.520171 12289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:46.520804 12289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:46.522014 12289 hybrid_clock.cc:648] HybridClock initialized: now 1790606506521985 us; error 29 us; skew 500 ppm
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606506, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:46.524585 12289 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:41:46.525352 12289 webserver.cc:460] Webserver started at http://127.1.201.2:42031/ using document root <none> and password file <none>
I20260928 14:41:46.525908 12289 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:46.525972 12289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:46.526207 12289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:46.528117 12289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "68551ca9df3d4cce85412ef9884361f9"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "fcbcb0a753a41a649bc0d1de48da1003"
server_key_iv: "eebecc2c7fbda6d9e892cb0be55e434c"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.528661 12289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "68551ca9df3d4cce85412ef9884361f9"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "fcbcb0a753a41a649bc0d1de48da1003"
server_key_iv: "eebecc2c7fbda6d9e892cb0be55e434c"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.532141 12289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:41:46.534614 12304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:46.535838 12289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:41:46.535948 12289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "68551ca9df3d4cce85412ef9884361f9"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "fcbcb0a753a41a649bc0d1de48da1003"
server_key_iv: "eebecc2c7fbda6d9e892cb0be55e434c"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.536043 12289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:46.547528 12289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:46.550277 12289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:46.550454 12289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:46.551076 12289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:46.552075 12289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:46.552155 12289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:46.552239 12289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:46.552261 12289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:46.560706 12289 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:45747
I20260928 14:41:46.560724 12417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:45747 every 8 connection(s)
I20260928 14:41:46.561579 12289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:41:46.563570 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 12289
I20260928 14:41:46.563707 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:41:46.563997 1828 external_mini_cluster.cc:1428] Setting key d6969a8d798e304eb1eafbf462f03a29
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606506, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
I20260928 14:41:46.575474 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.563358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:39065 (local address 127.1.201.62:37751)
0928 14:41:46.563517 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.563522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.564446 (+ 924us) server_negotiation.cc:407] Connection header received
0928 14:41:46.566129 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.566134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.566190 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.566266 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:46.567378 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.567876 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.568945 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.569210 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.569320 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.571975 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:46.571979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:46.571982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:46.572025 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:46.573411 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:46.573963 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:46.573967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:46.573969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:46.574048 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:46.574397 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:46.574400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:46.574402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:46.574577 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:46.574673 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.575121 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.575292 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
I20260928 14:41:46.576284 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.563826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:37751 (local address 127.1.201.2:39065)
0928 14:41:46.564297 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:46.564348 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:46.565934 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:46.566393 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:46.566398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:46.566741 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:46.567234 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.567249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.567993 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.567996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.568821 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.568831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.569356 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.569360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.569526 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.570133 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:46.570149 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:46.571790 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:46.573529 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:46.573535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:46.573550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:46.573863 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:46.574139 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:46.574141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:46.574143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:46.574274 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:46.574677 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:46.574684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:46.574908 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 14:41:46.575071 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 14:41:46.575392 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":126,"threads_started":1}
I20260928 14:41:46.577412 12418 heartbeater.cc:347] Connected to a master server at 127.1.201.62:37751
I20260928 14:41:46.577629 12418 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:46.578140 12418 heartbeater.cc:511] Master 127.1.201.62:37751 requested a full tablet report, sending...
I20260928 14:41:46.579123 12093 ts_manager.cc:194] Registered new tserver with Master: 68551ca9df3d4cce85412ef9884361f9 (127.1.201.2:45747)
I20260928 14:41:46.579689 12093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:39065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:41:46.590657 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:37751
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:41:46.694644 12423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:41:46.694873 12423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:41:46.694929 12423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:41:46.698874 12423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:41:46.698927 12423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:41:46.699029 12423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:41:46.704142 12423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:37751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:41:46.705158 12423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:41:46.706230 12423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:41:46.715974 12431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:46.715989 12428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:41:46.716106 12429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:41:46.716358 12423 server_base.cc:1034] running on GCE node
I20260928 14:41:46.716718 12423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:41:46.717293 12423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:41:46.718458 12423 hybrid_clock.cc:648] HybridClock initialized: now 1790606506718445 us; error 25 us; skew 500 ppm
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606506, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:41:46.721190 12423 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:41:46.722111 12423 webserver.cc:460] Webserver started at http://127.1.201.3:41197/ using document root <none> and password file <none>
I20260928 14:41:46.722615 12423 fs_manager.cc:359] Metadata directory not provided
I20260928 14:41:46.722675 12423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:41:46.722833 12423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:41:46.724767 12423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "7d682bda672e4ae5b69c484132bb2334"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "6ae58740570c057233450b298b7e1d0d"
server_key_iv: "f81cbaa2ac75d57c7c725873bdaba5ab"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.725255 12423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "7d682bda672e4ae5b69c484132bb2334"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "6ae58740570c057233450b298b7e1d0d"
server_key_iv: "f81cbaa2ac75d57c7c725873bdaba5ab"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.729120 12423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:41:46.731918 12438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:46.733503 12423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:41:46.733603 12423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "7d682bda672e4ae5b69c484132bb2334"
format_stamp: "Formatted at 2026-09-28 14:41:46 on dist-test-slave-0n27"
server_key: "6ae58740570c057233450b298b7e1d0d"
server_key_iv: "f81cbaa2ac75d57c7c725873bdaba5ab"
server_key_version: "encryptionkey@0"
I20260928 14:41:46.733727 12423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:41:46.744429 12423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:41:46.747704 12423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:41:46.747889 12423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:41:46.748474 12423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:41:46.749259 12423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:41:46.749305 12423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:46.749373 12423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:41:46.749415 12423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:41:46.757979 12423 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:38519
I20260928 14:41:46.758010 12551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:38519 every 8 connection(s)
I20260928 14:41:46.758812 12423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:41:46.766369 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 12423
I20260928 14:41:46.766501 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:41:46.766814 1828 external_mini_cluster.cc:1428] Setting key 40cfad6a7d262f58196f2103a1543727
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606506, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:41:46.771754 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.760412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:35135 (local address 127.1.201.62:37751)
0928 14:41:46.760560 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.760565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.761421 (+ 856us) server_negotiation.cc:407] Connection header received
0928 14:41:46.762796 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.762812 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.762875 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.762947 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:46.764076 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.764575 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.765821 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.766042 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.766139 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.768827 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:46.768831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:46.768835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:46.768862 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:46.770100 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:46.770531 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:46.770533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:46.770534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:46.770600 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:46.770900 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:46.770902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:46.770903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:46.771062 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:46.771133 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.771554 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.771640 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:41:46.772540 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.760846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:37751 (local address 127.1.201.3:35135)
0928 14:41:46.761271 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:46.761312 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:46.762549 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:46.763103 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:46.763109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:46.763441 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:41:46.763957 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.763973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.764681 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.764683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.765675 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.765684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.766203 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.766210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.766431 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.767071 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:41:46.767091 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:41:46.768665 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:41:46.770198 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:46.770202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:46.770213 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:46.770451 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:46.770673 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:41:46.770676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:41:46.770677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:41:46.770818 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:41:46.771145 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:41:46.771150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:41:46.771365 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 14:41:46.771508 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 14:41:46.771735 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":99,"threads_started":1}
I20260928 14:41:46.773507 12552 heartbeater.cc:347] Connected to a master server at 127.1.201.62:37751
I20260928 14:41:46.773722 12552 heartbeater.cc:464] Registering TS with master...
I20260928 14:41:46.774180 12552 heartbeater.cc:511] Master 127.1.201.62:37751 requested a full tablet report, sending...
I20260928 14:41:46.775053 12093 ts_manager.cc:194] Registered new tserver with Master: 7d682bda672e4ae5b69c484132bb2334 (127.1.201.3:38519)
I20260928 14:41:46.775552 12093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:35135
I20260928 14:41:46.780586 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:41:46.788924 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.782709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.1.201.62:37751)
0928 14:41:46.782846 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.782849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.782863 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:41:46.782972 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.782974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.783008 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.783070 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:41:46.783757 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.784209 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.784910 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.785089 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.785154 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.786381 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:41:46.786383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:41:46.786386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:41:46.786406 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:41:46.787725 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:46.788052 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:46.788054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:46.788055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:46.788118 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:41:46.788314 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:41:46.788316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:41:46.788317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:41:46.788494 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:41:46.788584 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.788748 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.788825 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:41:46.791601 12093 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:41:46.794427 12093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:41:46.807554 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.802065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:45747 (local address 127.0.0.1:54302)
0928 14:41:46.802388 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:46.802458 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:46.802599 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:46.803323 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:46.803330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:46.803355 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:46.803487 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.803494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.804719 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.804725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.805905 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.805915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.807041 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.807044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.807155 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.807195 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 14:41:46.807299 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:41:46.807420 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260928 14:41:46.807580 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.802239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.1.201.2:45747)
0928 14:41:46.802605 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.802617 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.802643 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:41:46.802755 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.802760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.802967 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.803115 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:46.803621 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.804604 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.806036 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.806923 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.807025 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.807164 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.807345 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.807434 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":139,"threads_started":1}
I20260928 14:41:46.807976 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.803090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:38519 (local address 127.0.0.1:54018)
0928 14:41:46.803584 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:46.803597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:46.803691 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:46.804283 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:46.804286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:46.804300 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:46.804387 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.804391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.805517 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.805520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.806557 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.806566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.807595 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.807601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.807694 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.807715 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:41:46.807777 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 14:41:46.807841 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":19,"thread_start_us":97,"threads_started":1}
I20260928 14:41:46.808128 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.802777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:38847 (local address 127.0.0.1:48928)
0928 14:41:46.803436 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:46.803448 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:46.803537 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:46.804047 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:46.804050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:46.804067 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:46.804152 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.804157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.805437 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.805440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.806581 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.806591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.807767 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.807770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.807857 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.807881 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:41:46.807950 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:41:46.808011 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":26,"thread_start_us":87,"threads_started":1}
I20260928 14:41:46.808359 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.803476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.1.201.3:38519)
0928 14:41:46.803872 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.803884 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.803902 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:41:46.803970 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.803977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.804136 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.804309 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:46.804449 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.805395 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.806679 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.807345 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.807427 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.807537 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.808111 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.808231 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":144,"threads_started":1}
I20260928 14:41:46.808475 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.802917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.1.201.1:38847)
0928 14:41:46.803311 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.803322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.803636 (+ 314us) server_negotiation.cc:407] Connection header received
0928 14:41:46.803724 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.803729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.803904 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.804047 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:46.804267 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.805311 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.806716 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.807480 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.807570 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.807712 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.808279 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.808359 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":102,"threads_started":1}
I20260928 14:41:46.810487 12352 tablet_service.cc:1467] Processing CreateTablet for tablet a19781bf4f6e4857a131fe5e4d8d1398 (DEFAULT_TABLE table=test-table [id=026ac9eb597e469fa2764b6c3e658d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:46.810968 12218 tablet_service.cc:1467] Processing CreateTablet for tablet a19781bf4f6e4857a131fe5e4d8d1398 (DEFAULT_TABLE table=test-table [id=026ac9eb597e469fa2764b6c3e658d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:46.811501 12352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19781bf4f6e4857a131fe5e4d8d1398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:46.811584 12486 tablet_service.cc:1467] Processing CreateTablet for tablet a19781bf4f6e4857a131fe5e4d8d1398 (DEFAULT_TABLE table=test-table [id=026ac9eb597e469fa2764b6c3e658d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:41:46.811959 12218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19781bf4f6e4857a131fe5e4d8d1398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:46.812542 12486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19781bf4f6e4857a131fe5e4d8d1398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:41:46.816990 12569 tablet_bootstrap.cc:492] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: Bootstrap starting.
I20260928 14:41:46.818885 12571 tablet_bootstrap.cc:492] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: Bootstrap starting.
I20260928 14:41:46.819190 12570 tablet_bootstrap.cc:492] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa: Bootstrap starting.
I20260928 14:41:46.819796 12569 tablet_bootstrap.cc:654] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:46.820627 12569 log.cc:824] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:46.821352 12570 tablet_bootstrap.cc:654] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:46.821455 12571 tablet_bootstrap.cc:654] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: Neither blocks nor log segments found. Creating new log.
I20260928 14:41:46.822153 12571 log.cc:824] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:46.822151 12570 log.cc:824] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa: Log is configured to *not* fsync() on all Append() calls
I20260928 14:41:46.822757 12569 tablet_bootstrap.cc:492] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: No bootstrap required, opened a new log
I20260928 14:41:46.822924 12569 ts_tablet_manager.cc:1397] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 14:41:46.823902 12571 tablet_bootstrap.cc:492] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: No bootstrap required, opened a new log
I20260928 14:41:46.823900 12570 tablet_bootstrap.cc:492] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa: No bootstrap required, opened a new log
I20260928 14:41:46.824052 12571 ts_tablet_manager.cc:1397] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:41:46.824052 12570 ts_tablet_manager.cc:1397] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:41:46.825465 12569 raft_consensus.cc:348] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 } } peers { permanent_uuid: "154158dbd03943bda8c0dae8bd268afa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38847 } } peers { permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 } }
I20260928 14:41:46.825644 12569 raft_consensus.cc:374] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:46.825681 12569 raft_consensus.cc:729] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68551ca9df3d4cce85412ef9884361f9, State: Initialized, Role: FOLLOWER
I20260928 14:41:46.826088 12569 consensus_queue.cc:260] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 } } peers { permanent_uuid: "154158dbd03943bda8c0dae8bd268afa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38847 } } peers { permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 } }
I20260928 14:41:46.826654 12418 heartbeater.cc:503] Master 127.1.201.62:37751 was elected leader, sending a full tablet report...
I20260928 14:41:46.826957 12569 ts_tablet_manager.cc:1428] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 14:41:46.827431 12571 raft_consensus.cc:348] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 } } peers { permanent_uuid: "154158dbd03943bda8c0dae8bd268afa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38847 } } peers { permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 } }
I20260928 14:41:46.827673 12571 raft_consensus.cc:374] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:46.827724 12571 raft_consensus.cc:729] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d682bda672e4ae5b69c484132bb2334, State: Initialized, Role: FOLLOWER
I20260928 14:41:46.827910 12570 raft_consensus.cc:348] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 } } peers { permanent_uuid: "154158dbd03943bda8c0dae8bd268afa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38847 } } peers { permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 } }
I20260928 14:41:46.828152 12570 raft_consensus.cc:374] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:41:46.828114 12571 consensus_queue.cc:260] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 } } peers { permanent_uuid: "154158dbd03943bda8c0dae8bd268afa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38847 } } peers { permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 } }
I20260928 14:41:46.828199 12570 raft_consensus.cc:729] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154158dbd03943bda8c0dae8bd268afa, State: Initialized, Role: FOLLOWER
I20260928 14:41:46.828539 12570 consensus_queue.cc:260] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 } } peers { permanent_uuid: "154158dbd03943bda8c0dae8bd268afa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38847 } } peers { permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 } }
I20260928 14:41:46.829011 12571 ts_tablet_manager.cc:1428] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:41:46.829347 12552 heartbeater.cc:503] Master 127.1.201.62:37751 was elected leader, sending a full tablet report...
I20260928 14:41:46.829388 12570 ts_tablet_manager.cc:1428] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:41:46.829648 12284 heartbeater.cc:503] Master 127.1.201.62:37751 was elected leader, sending a full tablet report...
I20260928 14:41:46.846971 12577 raft_consensus.cc:482] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:41:46.847113 12577 raft_consensus.cc:504] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 } } peers { permanent_uuid: "154158dbd03943bda8c0dae8bd268afa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38847 } } peers { permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 } }
I20260928 14:41:46.848178 12577 leader_election.cc:288] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68551ca9df3d4cce85412ef9884361f9 (127.1.201.2:45747), 7d682bda672e4ae5b69c484132bb2334 (127.1.201.3:38519)
I20260928 14:41:46.851544 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.848379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:45747 (local address 127.1.201.1:59931)
0928 14:41:46.848518 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:46.848533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:46.848618 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:46.848865 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:46.848868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:46.848882 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:46.848988 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.848995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.849648 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.849652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.850447 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.850456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.851121 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.851124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.851243 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.851269 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:41:46.851355 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:41:46.851421 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:41:46.851544 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.848453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:59931 (local address 127.1.201.2:45747)
0928 14:41:46.848572 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.848576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.848589 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:41:46.848747 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.848750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.848795 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.848858 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:46.849091 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.849549 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.850565 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.851000 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.851111 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.851171 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.851370 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.851439 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:41:46.851958 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.848724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:40103 (local address 127.1.201.3:38519)
0928 14:41:46.848844 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.848850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.849044 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:41:46.849153 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.849157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.849198 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.849268 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:41:46.849512 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.850022 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.851030 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.851425 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.851539 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.851591 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.851780 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.851844 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:41:46.851969 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.848647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:38519 (local address 127.1.201.1:40103)
0928 14:41:46.848963 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:46.848973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:41:46.849079 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:41:46.849288 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:41:46.849291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:41:46.849305 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:41:46.849418 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.849422 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.850122 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.850126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.850913 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:41:46.850921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.851544 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.851547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:41:46.851652 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.851673 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:41:46.851765 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:41:46.851830 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
I20260928 14:41:46.852212 12369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19781bf4f6e4857a131fe5e4d8d1398" candidate_uuid: "154158dbd03943bda8c0dae8bd268afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68551ca9df3d4cce85412ef9884361f9" is_pre_election: true
I20260928 14:41:46.852489 12369 raft_consensus.cc:2454] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 154158dbd03943bda8c0dae8bd268afa in term 0.
I20260928 14:41:46.852887 12171 leader_election.cc:302] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154158dbd03943bda8c0dae8bd268afa, 68551ca9df3d4cce85412ef9884361f9; no voters:
I20260928 14:41:46.853135 12577 raft_consensus.cc:2781] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:41:46.853235 12577 raft_consensus.cc:482] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:41:46.853279 12577 raft_consensus.cc:3037] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:46.853204 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19781bf4f6e4857a131fe5e4d8d1398" candidate_uuid: "154158dbd03943bda8c0dae8bd268afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d682bda672e4ae5b69c484132bb2334" is_pre_election: true
I20260928 14:41:46.853458 12506 raft_consensus.cc:2454] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 154158dbd03943bda8c0dae8bd268afa in term 0.
I20260928 14:41:46.854427 12577 raft_consensus.cc:504] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 } } peers { permanent_uuid: "154158dbd03943bda8c0dae8bd268afa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38847 } } peers { permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 } }
I20260928 14:41:46.854772 12577 leader_election.cc:288] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [CANDIDATE]: Term 1 election: Requested vote from peers 68551ca9df3d4cce85412ef9884361f9 (127.1.201.2:45747), 7d682bda672e4ae5b69c484132bb2334 (127.1.201.3:38519)
I20260928 14:41:46.855069 12369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19781bf4f6e4857a131fe5e4d8d1398" candidate_uuid: "154158dbd03943bda8c0dae8bd268afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68551ca9df3d4cce85412ef9884361f9"
I20260928 14:41:46.855139 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19781bf4f6e4857a131fe5e4d8d1398" candidate_uuid: "154158dbd03943bda8c0dae8bd268afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d682bda672e4ae5b69c484132bb2334"
I20260928 14:41:46.855193 12369 raft_consensus.cc:3037] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:46.855257 12506 raft_consensus.cc:3037] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:41:46.856184 12506 raft_consensus.cc:2454] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154158dbd03943bda8c0dae8bd268afa in term 1.
I20260928 14:41:46.856195 12369 raft_consensus.cc:2454] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 154158dbd03943bda8c0dae8bd268afa in term 1.
I20260928 14:41:46.856446 12171 leader_election.cc:302] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154158dbd03943bda8c0dae8bd268afa, 68551ca9df3d4cce85412ef9884361f9; no voters:
I20260928 14:41:46.856607 12577 raft_consensus.cc:2781] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:41:46.856827 12577 raft_consensus.cc:686] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [term 1 LEADER]: Becoming Leader. State: Replica: 154158dbd03943bda8c0dae8bd268afa, State: Running, Role: LEADER
I20260928 14:41:46.857132 12577 consensus_queue.cc:237] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 } } peers { permanent_uuid: "154158dbd03943bda8c0dae8bd268afa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38847 } } peers { permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 } }
I20260928 14:41:46.860021 12092 catalog_manager.cc:5495] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa reported cstate change: term changed from 0 to 1, leader changed from <none> to 154158dbd03943bda8c0dae8bd268afa (127.1.201.1). New cstate: current_term: 1 leader_uuid: "154158dbd03943bda8c0dae8bd268afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "154158dbd03943bda8c0dae8bd268afa" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 } health_report { overall_health: UNKNOWN } } }
W20260928 14:41:46.869194 12285 tablet.cc:2369] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:41:46.873673 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.870790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.1.201.1:38847)
0928 14:41:46.870931 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:41:46.870934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:41:46.870946 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:41:46.871048 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:41:46.871050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:41:46.871096 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:41:46.871179 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:41:46.871370 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.871839 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.872589 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:41:46.872775 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:41:46.872838 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:41:46.873054 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:41:46.873394 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:41:46.873472 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:41:46.873523 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:41:46.873565 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:41:46.879027 12506 raft_consensus.cc:1261] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334 [term 1 FOLLOWER]: Refusing update from remote peer 154158dbd03943bda8c0dae8bd268afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:46.879031 12369 raft_consensus.cc:1261] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9 [term 1 FOLLOWER]: Refusing update from remote peer 154158dbd03943bda8c0dae8bd268afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:41:46.879640 12577 consensus_queue.cc:1035] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "68551ca9df3d4cce85412ef9884361f9" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:46.879828 12579 consensus_queue.cc:1035] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d682bda672e4ae5b69c484132bb2334" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 38519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:41:46.887836 12583 mvcc.cc:205] Tried to move back new op lower bound from 7334324252170129408 to 7334324252089028608. Current Snapshot: MvccSnapshot[applied={T|T < 7334324252170129408}]
I20260928 14:41:46.894568 12088 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:46.894770 12088 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:41:46.896970 12088 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76d5f7145e304d63a88637e714cfc317: Sending DeleteTablet for 3 replicas of tablet a19781bf4f6e4857a131fe5e4d8d1398
I20260928 14:41:46.897899 12218 tablet_service.cc:1514] Processing DeleteTablet for tablet a19781bf4f6e4857a131fe5e4d8d1398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:46 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:48928
I20260928 14:41:46.897905 12352 tablet_service.cc:1514] Processing DeleteTablet for tablet a19781bf4f6e4857a131fe5e4d8d1398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:46 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:54302
I20260928 14:41:46.898370 12592 tablet_replica.cc:330] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: stopping tablet replica
I20260928 14:41:46.898371 12486 tablet_service.cc:1514] Processing DeleteTablet for tablet a19781bf4f6e4857a131fe5e4d8d1398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:41:46 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:54018
I20260928 14:41:46.898649 12593 tablet_replica.cc:330] T a19781bf4f6e4857a131fe5e4d8d1398 P 154158dbd03943bda8c0dae8bd268afa: stopping tablet replica
I20260928 14:41:46.898677 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 12155
I20260928 14:41:46.899411 12592 raft_consensus.cc:2229] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:46.899766 12592 raft_consensus.cc:2258] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:46.900324 12594 tablet_replica.cc:330] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: stopping tablet replica
I20260928 14:41:46.900589 12594 raft_consensus.cc:2229] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:41:46.900862 12594 raft_consensus.cc:2258] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:41:46.901262 12592 ts_tablet_manager.cc:1905] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:46.902078 12594 ts_tablet_manager.cc:1905] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:41:46.904062 12594 ts_tablet_manager.cc:1918] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:46.904178 12594 log.cc:1196] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/a19781bf4f6e4857a131fe5e4d8d1398
I20260928 14:41:46.904474 12594 ts_tablet_manager.cc:1939] T a19781bf4f6e4857a131fe5e4d8d1398 P 7d682bda672e4ae5b69c484132bb2334: Deleting consensus metadata
I20260928 14:41:46.904712 12592 ts_tablet_manager.cc:1918] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:41:46.904811 12592 log.cc:1196] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/a19781bf4f6e4857a131fe5e4d8d1398
I20260928 14:41:46.905146 12592 ts_tablet_manager.cc:1939] T a19781bf4f6e4857a131fe5e4d8d1398 P 68551ca9df3d4cce85412ef9884361f9: Deleting consensus metadata
I20260928 14:41:46.905622 12077 catalog_manager.cc:4841] TS 7d682bda672e4ae5b69c484132bb2334 (127.1.201.3:38519): tablet a19781bf4f6e4857a131fe5e4d8d1398 (table test-table [id=026ac9eb597e469fa2764b6c3e658d0f]) successfully deleted
W20260928 14:41:46.906330 12077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:38847 (error 108)
I20260928 14:41:46.906378 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 12289
I20260928 14:41:46.906837 12077 catalog_manager.cc:4841] TS 68551ca9df3d4cce85412ef9884361f9 (127.1.201.2:45747): tablet a19781bf4f6e4857a131fe5e4d8d1398 (table test-table [id=026ac9eb597e469fa2764b6c3e658d0f]) successfully deleted
I20260928 14:41:46.910554 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:41:46.910303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:38847 (local address 127.0.0.1:48950)
0928 14:41:46.910426 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 14:41:46.910484 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:38847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 14:41:46.910787 12077 catalog_manager.cc:4584] TS 154158dbd03943bda8c0dae8bd268afa (127.1.201.1:38847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a19781bf4f6e4857a131fe5e4d8d1398: Network error: Client connection negotiation failed: client connection to 127.1.201.1:38847: connect: Connection refused (error 111)
I20260928 14:41:46.912057 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 12423
I20260928 14:41:46.916451 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 12061
2026-09-28T14:41:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2082 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12598](info): set up 2 sockets
Sep 28 14:41:46 dist-test-slave-0n27 krb5kdc[12598](info): commencing operation
krb5kdc: starting...
W20260928 14:41:47.887392 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:41:47 dist-test-slave-0n27 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:41:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:41:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.201.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.201.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/rangeradmin_127.1.201.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/rangeradmin_127.1.201.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.201.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.201.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/rangerlookup_127.1.201.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/rangerlookup_127.1.201.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.201.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.201.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/HTTP_127.1.201.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/HTTP_127.1.201.20@KRBTEST.COM.keytab.
I20260928 14:41:48.040437 1828 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskScXicz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/postgres -l logfile start
2026-09-28 14:41:50.346 UTC [12628] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:41:50.346 UTC [12628] LOG: listening on IPv4 address "127.1.201.20", port 47763
2026-09-28 14:41:50.349 UTC [12628] LOG: listening on Unix socket "/tmp/.s.PGSQL.47763"
2026-09-28 14:41:50.369 UTC [12631] LOG: database system was shut down at 2026-09-28 14:41:49 UTC
2026-09-28 14:41:50.372 UTC [12628] LOG: database system is ready to accept connections
I20260928 14:41:51.263320 1828 mini_postgres.cc:96] Postgres bound to 47763
2026-09-28 14:41:51.269 UTC [12639] FATAL: database "slave" does not exist
127.1.201.20:47763 - accepting connections
I20260928 14:41:51.270468 1828 mini_ranger.cc:161] Starting Ranger...
I20260928 14:41:51.285252 1828 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:41:51.734958 1828 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:41:51.735038 1828 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-admin
2026-09-28 14:41:52,017 [I] DB FLAVOR :POSTGRES
2026-09-28 14:41:52,017 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:41:52,017 [I] Checking connection..
2026-09-28 14:41:52,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:41:52,272 [I] Checking connection passed.
2026-09-28 14:41:52,273 [I] --------- Verifying version history table ---------
2026-09-28 14:41:52,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:41:52,526 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:41:52,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:41:52,756 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:41:52,757 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:41:52,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:41:52.980 UTC [12703] WARNING: there is no transaction in progress
2026-09-28 14:41:52,986 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:41:52,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:41:53,236 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:41:53,236 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:41:53,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:41:53,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:41:53,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0n27.c.gcp-upstream.internal','N') ;"
2026-09-28 14:41:53,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:41:54,190 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:41:54,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:41:54,441 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:41:54,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:41:54,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:41:54,934 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:41:54,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:41:55.850 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:55.866 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:55.881 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:55.895 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:55.910 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:55.927 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:55.989 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:55.995 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:56.008 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:56.018 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:56.027 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:56.040 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:56.050 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:56.056 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:56.062 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:56.068 UTC [12811] WARNING: there is no transaction in progress
2026-09-28 14:41:56,645 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:41:56,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:41:56,885 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:41:56,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:41:57,140 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:41:57,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:41:57,372 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:41:57,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:41:57,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:41:57,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0n27.c.gcp-upstream.internal';"
2026-09-28 14:41:58,100 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:41:58,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:41:58,330 [I] DB_PATCHES have already been applied
I20260928 14:41:58.336287 1828 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-admin:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskScXicz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:41:58.336349 1828 mini_ranger.cc:193] Using host: 127.1.201.20
I20260928 14:41:58.339998 1828 mini_ranger.cc:227] Ranger admin URL: http://127.1.201.20:37113
Sep 28, 2026 2:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:41:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:41:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:41:59.406172 1828 mini_ranger.cc:160] Time spent starting Ranger: real 8.136s user 0.000s sys 0.011s
Sep 28, 2026 2:42:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:42:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 2:42:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:42:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:42:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:42:20.367714 1828 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.201.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.201.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/rangerkms_127.1.201.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/rangerkms_127.1.201.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.201.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.201.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.201.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/HTTP_127.1.201.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.201.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/HTTP_127.1.201.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:42:20 dist-test-slave-0n27 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606540, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:42:20.447847 1828 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:42:20.456959 1828 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:42:21.222605 1828 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:42:21.222721 1828 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-kms
I20260928 14:42:21.222821 1828 mini_ranger_kms.cc:227] Using postgres at 127.1.201.20:47763
2026-09-28 14:42:21,612 [I] DB FLAVOR :POSTGRES
2026-09-28 14:42:21,612 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:42:21,613 [I] Checking connection
2026-09-28 14:42:21,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:42:21,866 [I] connection success
2026-09-28 14:42:21,867 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:42:21,867 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:42:21,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:42:22,094 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:42:22,095 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:42:22,095 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:42:22,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:47763/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:42:22,335 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:42:22.777627 1828 mini_ranger_kms.cc:310] Created kms service
I20260928 14:42:22.992810 1828 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:42:23.035162 1828 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:42:23 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606540, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:42:23.423085 1828 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:42:23.423247 1828 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-kms:/tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:42:23.423271 1828 mini_ranger_kms.cc:240] Using host: 127.1.201.20
I20260928 14:42:23.427199 1828 mini_ranger_kms.cc:276] Ranger KMS PID: 12991
I20260928 14:42:23.427337 1828 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.201.20:51905
Sep 28, 2026 2:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:42:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:42:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:42:25.585021 1828 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.137s user 0.019s sys 0.182s
I20260928 14:42:25.585150 1828 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:42:25.585261 1828 mini_ranger_kms.cc:401] 127.1.201.20:51905/kms/v1/keys
Sep 28 14:42:25 dist-test-slave-0n27 krb5kdc[12598](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606545, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.201.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:42:26 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606545, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.201.20@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
Sep 28 14:42:26 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606545, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.201.20@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
Sep 28 14:42:28 dist-test-slave-0n27 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:42:28.785156 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:35787
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:35787
--ranger_config_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:51905/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:42:28.910379 13042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:42:28.910676 13042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:42:28.910717 13042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:42:28.914844 13042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:42:28.914907 13042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:42:28.914927 13042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:42:28.914943 13042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:42:28.914958 13042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:42:28.920625 13042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.201.20:51905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:35787
--ranger_config_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:35787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:42:28.921754 13042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:42:28.922914 13042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:42:28.931610 13047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:42:28.931612 13048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:42:28.931813 13042 server_base.cc:1034] running on GCE node
W20260928 14:42:28.931967 13050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:42:28.932539 13042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:42:28.933748 13042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:42:28.934947 13042 hybrid_clock.cc:648] HybridClock initialized: now 1790606548934917 us; error 40 us; skew 500 ppm
Sep 28 14:42:28 dist-test-slave-0n27 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606548, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:42:28.938236 13042 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:42:28.939226 13042 webserver.cc:460] Webserver started at http://127.1.201.62:44353/ using document root <none> and password file <none>
I20260928 14:42:28.939792 13042 fs_manager.cc:359] Metadata directory not provided
I20260928 14:42:28.939845 13042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:42:28.940068 13042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:42:28 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606548, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:42:28.993556 13042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "2bd03dea082244b580927d6f5a927928"
format_stamp: "Formatted at 2026-09-28 14:42:28 on dist-test-slave-0n27"
server_key: "9a6d8e4df5ac2dc242a9a090b50c044c"
server_key_iv: "2a4089232f045b2c5ed7773a63b3deeb"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:28.994309 13042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "2bd03dea082244b580927d6f5a927928"
format_stamp: "Formatted at 2026-09-28 14:42:28 on dist-test-slave-0n27"
server_key: "9a6d8e4df5ac2dc242a9a090b50c044c"
server_key_iv: "2a4089232f045b2c5ed7773a63b3deeb"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:28.998351 13042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:42:29.016602 13057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:42:29.018105 13042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:42:29.018242 13042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "2bd03dea082244b580927d6f5a927928"
format_stamp: "Formatted at 2026-09-28 14:42:28 on dist-test-slave-0n27"
server_key: "9a6d8e4df5ac2dc242a9a090b50c044c"
server_key_iv: "2a4089232f045b2c5ed7773a63b3deeb"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:29.018373 13042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:42:29.030462 13042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:42:29.033434 13042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:42:29.033627 13042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:42:29.042297 13042 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:35787
I20260928 14:42:29.042346 13109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:35787 every 8 connection(s)
I20260928 14:42:29.043278 13042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:42:29.046129 13110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:42:29.051769 13110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928: Bootstrap starting.
I20260928 14:42:29.053620 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 13042
I20260928 14:42:29.053743 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:42:29.054109 1828 external_mini_cluster.cc:1428] Setting key b047a467df8607e868838aba9f262e66
I20260928 14:42:29.054363 13110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928: Neither blocks nor log segments found. Creating new log.
I20260928 14:42:29.055317 13110 log.cc:824] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928: Log is configured to *not* fsync() on all Append() calls
I20260928 14:42:29.057572 13110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928: No bootstrap required, opened a new log
I20260928 14:42:29.061849 13110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd03dea082244b580927d6f5a927928" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35787 } }
I20260928 14:42:29.062211 13110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:42:29.062351 13110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd03dea082244b580927d6f5a927928, State: Initialized, Role: FOLLOWER
I20260928 14:42:29.062886 13110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd03dea082244b580927d6f5a927928" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35787 } }
I20260928 14:42:29.063071 13110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:42:29.063184 13110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:42:29.063303 13110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:42:29.064419 13110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd03dea082244b580927d6f5a927928" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35787 } }
Sep 28 14:42:29 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:42:29.064810 13110 leader_election.cc:302] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bd03dea082244b580927d6f5a927928; no voters:
I20260928 14:42:29.065164 13110 leader_election.cc:288] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:42:29.065235 13115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:42:29.065516 13115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [term 1 LEADER]: Becoming Leader. State: Replica: 2bd03dea082244b580927d6f5a927928, State: Running, Role: LEADER
I20260928 14:42:29.066175 13115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd03dea082244b580927d6f5a927928" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35787 } }
I20260928 14:42:29.066629 13110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:42:29.067644 13115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd03dea082244b580927d6f5a927928. Latest consensus state: current_term: 1 leader_uuid: "2bd03dea082244b580927d6f5a927928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd03dea082244b580927d6f5a927928" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35787 } } }
I20260928 14:42:29.067853 13115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [sys.catalog]: This master's current role is: LEADER
I20260928 14:42:29.068269 13124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:42:29.070994 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:29.055337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.1.201.62:35787)
0928 14:42:29.055949 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:29.055964 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:29.055998 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:42:29.056806 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:29.056830 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:29.057215 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:29.057557 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:42:29.058445 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:29.060062 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:29.061300 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:29.062023 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:29.062196 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:29.065362 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:42:29.065369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:42:29.065387 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:42:29.065417 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:42:29.067361 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:42:29.067874 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:42:29.067879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:42:29.067885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:42:29.068011 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:42:29.068308 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:42:29.068311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:42:29.068313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:42:29.068799 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:42:29.068977 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:29.069702 (+ 725us) server_negotiation.cc:299] Negotiation successful
0928 14:42:29.070025 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":148,"threads_started":1}
I20260928 14:42:29.072520 13124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:42:29.072911 13110 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:42:29.073984 13116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bd03dea082244b580927d6f5a927928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd03dea082244b580927d6f5a927928" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 35787 } } }
I20260928 14:42:29.074080 13116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928 [sys.catalog]: This master's current role is: LEADER
I20260928 14:42:29.083377 13124 catalog_manager.cc:1347] Generated new cluster ID: 5b75231a42ff47a59e3dd095f6aa50b6
I20260928 14:42:29.083456 13124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:42:29.106442 13124 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:42:29.107818 13124 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:42:29.116456 13124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928: Generated new TSK 0
I20260928 14:42:29.117345 13124 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:42:29.825507 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:51905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35787
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:42:29.967270 13155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:42:29.967700 13155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:42:29.967803 13155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:42:29.972132 13155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:42:29.972211 13155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:42:29.972347 13155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:42:29.977737 13155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.201.20:51905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:42:29.979060 13155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:42:29.980274 13155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:42:29.989691 13162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:42:29.989965 13165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:42:29.990209 13155 server_base.cc:1034] running on GCE node
W20260928 14:42:29.990357 13163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:42:29.990788 13155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:42:29.991403 13155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:42:29.992609 13155 hybrid_clock.cc:648] HybridClock initialized: now 1790606549992574 us; error 46 us; skew 500 ppm
Sep 28 14:42:29 dist-test-slave-0n27 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606549, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:42:30.004227 13155 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:42:30.005236 13155 webserver.cc:460] Webserver started at http://127.1.201.1:42165/ using document root <none> and password file <none>
I20260928 14:42:30.005882 13155 fs_manager.cc:359] Metadata directory not provided
I20260928 14:42:30.005975 13155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:42:30.006208 13155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:42:30 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606549, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:42:30.025904 13155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "90194cc139d14f4abccfc385117df615"
format_stamp: "Formatted at 2026-09-28 14:42:30 on dist-test-slave-0n27"
server_key: "17ab387847bc7dddec394d317fe91928"
server_key_iv: "224bc7bde04050c52e80524777ee0139"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:30.026748 13155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "90194cc139d14f4abccfc385117df615"
format_stamp: "Formatted at 2026-09-28 14:42:30 on dist-test-slave-0n27"
server_key: "17ab387847bc7dddec394d317fe91928"
server_key_iv: "224bc7bde04050c52e80524777ee0139"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:30.037449 13155 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260928 14:42:30.062678 13172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:42:30.064417 13155 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:42:30.064546 13155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "90194cc139d14f4abccfc385117df615"
format_stamp: "Formatted at 2026-09-28 14:42:30 on dist-test-slave-0n27"
server_key: "17ab387847bc7dddec394d317fe91928"
server_key_iv: "224bc7bde04050c52e80524777ee0139"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:30.064695 13155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:42:30.099231 13155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:42:30.103366 13155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:42:30.103624 13155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:42:30.104382 13155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:42:30.105518 13155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:42:30.105623 13155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:42:30.105938 13155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:42:30.106052 13155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:42:30.117970 13155 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:42705
I20260928 14:42:30.119482 13155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:42:30.120491 13287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:42705 every 8 connection(s)
I20260928 14:42:30.124595 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 13155
I20260928 14:42:30.124720 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:42:30.125051 1828 external_mini_cluster.cc:1428] Setting key 3d8112526d9657f7c613671b55c33302
Sep 28 14:42:30 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606549, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:42:30.141613 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.124973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:45971 (local address 127.1.201.62:35787)
0928 14:42:30.126979 (+ 2006us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:30.126983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:30.126998 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:42:30.127041 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:30.127044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:30.127086 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:30.127173 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:42:30.128797 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.129624 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.131439 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.131792 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.131873 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.135147 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:42:30.135151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:42:30.135154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:42:30.135181 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:42:30.137240 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:42:30.137926 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:42:30.137932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:42:30.137937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:42:30.138041 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:42:30.140298 (+ 2257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:42:30.140302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:42:30.140303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:42:30.140532 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:42:30.140612 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:30.141313 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 14:42:30.141442 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1931,"thread_start_us":86,"threads_started":1}
I20260928 14:42:30.143508 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.123830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35787 (local address 127.1.201.1:45971)
0928 14:42:30.125502 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0928 14:42:30.125560 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:42:30.126575 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:42:30.127494 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:42:30.127506 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:42:30.127888 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:42:30.128598 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.128619 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.129856 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.129859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.131293 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.131304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.132080 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.132085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.132284 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.133013 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:42:30.133034 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:42:30.134946 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:42:30.137391 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:42:30.137400 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:42:30.137417 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:42:30.137805 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:42:30.138155 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:42:30.138159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:42:30.138161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:42:30.138341 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:42:30.140691 (+ 2350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:42:30.140697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:42:30.141032 (+ 335us) client_negotiation.cc:769] Sending connection context
0928 14:42:30.142104 (+ 1072us) client_negotiation.cc:240] Negotiation successful
0928 14:42:30.142446 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1450,"thread_start_us":146,"threads_started":1}
I20260928 14:42:30.146060 13288 heartbeater.cc:347] Connected to a master server at 127.1.201.62:35787
I20260928 14:42:30.146308 13288 heartbeater.cc:464] Registering TS with master...
I20260928 14:42:30.146893 13288 heartbeater.cc:511] Master 127.1.201.62:35787 requested a full tablet report, sending...
I20260928 14:42:30.148676 13074 ts_manager.cc:194] Registered new tserver with Master: 90194cc139d14f4abccfc385117df615 (127.1.201.1:42705)
I20260928 14:42:30.150405 13074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:45971
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
Sep 28 14:42:30 dist-test-slave-0n27 krb5kdc[12598](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:42:30.193760 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:51905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35787
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:42:30.359220 13294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:42:30.359669 13294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:42:30.359791 13294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:42:30.365692 13294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:42:30.365854 13294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:42:30.365993 13294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:42:30.374043 13294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.201.20:51905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:42:30.375620 13294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:42:30.376997 13294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:42:30.389974 13300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:42:30.390874 13303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:42:30.391173 13294 server_base.cc:1034] running on GCE node
W20260928 14:42:30.391383 13301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:42:30.391880 13294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:42:30.392623 13294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:42:30.393829 13294 hybrid_clock.cc:648] HybridClock initialized: now 1790606550393760 us; error 83 us; skew 500 ppm
Sep 28 14:42:30 dist-test-slave-0n27 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:42:30.397405 13294 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:42:30.398496 13294 webserver.cc:460] Webserver started at http://127.1.201.2:34405/ using document root <none> and password file <none>
I20260928 14:42:30.399192 13294 fs_manager.cc:359] Metadata directory not provided
I20260928 14:42:30.399266 13294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:42:30.399479 13294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:42:30 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:42:30.419661 13294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a"
format_stamp: "Formatted at 2026-09-28 14:42:30 on dist-test-slave-0n27"
server_key: "48f1d5e607ea4046ad8dde001621329f"
server_key_iv: "e6966a23f8733c49454d7df9f6f54e8b"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:30.420588 13294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a"
format_stamp: "Formatted at 2026-09-28 14:42:30 on dist-test-slave-0n27"
server_key: "48f1d5e607ea4046ad8dde001621329f"
server_key_iv: "e6966a23f8733c49454d7df9f6f54e8b"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:30.425796 13294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 14:42:30.444248 13310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:42:30.446185 13294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:42:30.446310 13294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a"
format_stamp: "Formatted at 2026-09-28 14:42:30 on dist-test-slave-0n27"
server_key: "48f1d5e607ea4046ad8dde001621329f"
server_key_iv: "e6966a23f8733c49454d7df9f6f54e8b"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:30.446432 13294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:42:30.477869 13294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:42:30.480908 13294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:42:30.481110 13294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:42:30.481684 13294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:42:30.482558 13294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:42:30.482627 13294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:42:30.482733 13294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:42:30.482795 13294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:42:30.492345 13294 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:39083
I20260928 14:42:30.493512 13423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:39083 every 8 connection(s)
I20260928 14:42:30.493595 13294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:42:30.495956 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 13294
I20260928 14:42:30.496068 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:42:30.496407 1828 external_mini_cluster.cc:1428] Setting key 62dbffcc2dc06a6c87a7f42a3c0b18b5
Sep 28 14:42:30 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
I20260928 14:42:30.512277 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.497001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:60915 (local address 127.1.201.62:35787)
0928 14:42:30.497155 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:30.497159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:30.499893 (+ 2734us) server_negotiation.cc:407] Connection header received
0928 14:42:30.500807 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:30.500810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:30.500861 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:30.500938 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:42:30.502537 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.503078 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.504276 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.504672 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.504764 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.507796 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:42:30.507802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:42:30.507808 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:42:30.507841 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:42:30.509659 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:42:30.510364 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:42:30.510367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:42:30.510370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:42:30.510452 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:42:30.510887 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:42:30.510892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:42:30.510896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:42:30.511139 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:42:30.511240 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:30.511962 (+ 722us) server_negotiation.cc:299] Negotiation successful
0928 14:42:30.512097 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:42:30.514309 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.496340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35787 (local address 127.1.201.2:60915)
0928 14:42:30.499753 (+ 3413us) negotiation.cc:107] Waiting for socket to connect
0928 14:42:30.499798 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:42:30.500602 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:42:30.501281 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:42:30.501292 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:42:30.501666 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:42:30.502389 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.502407 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.503201 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.503205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.504111 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.504120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.505003 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.505009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.505253 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.506007 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:42:30.506034 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:42:30.507589 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:42:30.509847 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:42:30.509855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:42:30.509872 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:42:30.510236 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:42:30.510572 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:42:30.510574 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:42:30.510577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:42:30.510761 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:42:30.511307 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:42:30.511313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:42:30.511671 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 14:42:30.512860 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0928 14:42:30.513208 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1366,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
I20260928 14:42:30.516429 13424 heartbeater.cc:347] Connected to a master server at 127.1.201.62:35787
I20260928 14:42:30.516674 13424 heartbeater.cc:464] Registering TS with master...
I20260928 14:42:30.517266 13424 heartbeater.cc:511] Master 127.1.201.62:35787 requested a full tablet report, sending...
I20260928 14:42:30.518437 13074 ts_manager.cc:194] Registered new tserver with Master: f7ddd0e0ed694d599a1f3fa09f88b38a (127.1.201.2:39083)
I20260928 14:42:30.519140 13074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:60915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:42:30.542022 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:51905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35787
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:42:30.711138 13429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:42:30.711583 13429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:42:30.711709 13429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:42:30.718016 13429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:42:30.718166 13429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:42:30.718302 13429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:42:30.726190 13429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.201.20:51905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:42:30.727802 13429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:42:30.729188 13429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 28 14:42:30 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
W20260928 14:42:30.744166 13437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:42:30.746141 13434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:42:30.746771 13435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:42:30.753657 13429 server_base.cc:1034] running on GCE node
I20260928 14:42:30.754328 13429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:42:30.756700 13429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:42:30.772717 13429 hybrid_clock.cc:648] HybridClock initialized: now 1790606550770164 us; error 2586 us; skew 500 ppm
Sep 28 14:42:30 dist-test-slave-0n27 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:42:30.777225 13429 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:42:30.778892 13429 webserver.cc:460] Webserver started at http://127.1.201.3:35011/ using document root <none> and password file <none>
I20260928 14:42:30.779680 13429 fs_manager.cc:359] Metadata directory not provided
I20260928 14:42:30.779930 13429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:42:30.780239 13429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:42:30 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:42:30.819939 13429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "9e40636a5782448287f18e5085f958ad"
format_stamp: "Formatted at 2026-09-28 14:42:30 on dist-test-slave-0n27"
server_key: "b7fb9ed00af13a6f3b295283b6182e53"
server_key_iv: "4f8628141cea4262114d8cfefaa3a688"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:30.820628 13429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "9e40636a5782448287f18e5085f958ad"
format_stamp: "Formatted at 2026-09-28 14:42:30 on dist-test-slave-0n27"
server_key: "b7fb9ed00af13a6f3b295283b6182e53"
server_key_iv: "4f8628141cea4262114d8cfefaa3a688"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:30.825852 13429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:42:30.841745 13444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:42:30.843278 13429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:42:30.843431 13429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "9e40636a5782448287f18e5085f958ad"
format_stamp: "Formatted at 2026-09-28 14:42:30 on dist-test-slave-0n27"
server_key: "b7fb9ed00af13a6f3b295283b6182e53"
server_key_iv: "4f8628141cea4262114d8cfefaa3a688"
server_key_version: "kuduclusterkey@0"
I20260928 14:42:30.843572 13429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:42:30.860760 13429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:42:30.863880 13429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:42:30.864077 13429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:42:30.864933 13429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:42:30.866001 13429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:42:30.866115 13429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:42:30.866211 13429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:42:30.866276 13429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:42:30.876654 13429 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:32955
I20260928 14:42:30.877856 13429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:42:30.878978 13557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:32955 every 8 connection(s)
I20260928 14:42:30.882805 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 13429
I20260928 14:42:30.882927 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:42:30.883253 1828 external_mini_cluster.cc:1428] Setting key 9dd1b4fa20db1045110378a99c320479
Sep 28 14:42:30 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:42:30.895051 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.880758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:55477 (local address 127.1.201.62:35787)
0928 14:42:30.880915 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:30.880920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:30.881971 (+ 1051us) server_negotiation.cc:407] Connection header received
0928 14:42:30.883244 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:30.883247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:30.883302 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:30.883354 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:42:30.884980 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.885847 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.887476 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.887828 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.887935 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.890795 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:42:30.890799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:42:30.890802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:42:30.890829 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:42:30.892508 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:42:30.893125 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:42:30.893129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:42:30.893131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:42:30.893221 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:42:30.893714 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:42:30.893718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:42:30.893720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:42:30.893989 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:42:30.894094 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:30.894748 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 14:42:30.894886 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:42:30.896140 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.881225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:35787 (local address 127.1.201.3:55477)
0928 14:42:30.881807 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 14:42:30.881866 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:42:30.882983 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:42:30.883756 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:42:30.883764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:42:30.884117 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:42:30.884792 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.884812 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.885956 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.885959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.887339 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.887351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.887998 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.888002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.888209 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.888906 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:42:30.888926 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:42:30.890627 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:42:30.892632 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:42:30.892638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:42:30.892655 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:42:30.892997 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:42:30.893375 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:42:30.893378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:42:30.893380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:42:30.893594 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:42:30.894128 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:42:30.894135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:42:30.894466 (+ 331us) client_negotiation.cc:769] Sending connection context
0928 14:42:30.894680 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 14:42:30.895016 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":146,"threads_started":1}
I20260928 14:42:30.897830 13558 heartbeater.cc:347] Connected to a master server at 127.1.201.62:35787
I20260928 14:42:30.898084 13558 heartbeater.cc:464] Registering TS with master...
I20260928 14:42:30.898645 13558 heartbeater.cc:511] Master 127.1.201.62:35787 requested a full tablet report, sending...
I20260928 14:42:30.899791 13074 ts_manager.cc:194] Registered new tserver with Master: 9e40636a5782448287f18e5085f958ad (127.1.201.3:32955)
I20260928 14:42:30.900523 13074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:55477
I20260928 14:42:30.909281 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:42:30.923741 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.914285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.1.201.62:35787)
0928 14:42:30.914409 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:30.914414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:30.914667 (+ 253us) server_negotiation.cc:407] Connection header received
0928 14:42:30.914818 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:30.914823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:30.914886 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:30.914979 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:42:30.915918 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.916721 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.917999 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.918338 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.918446 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.920112 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:42:30.920116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:42:30.920119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:42:30.920146 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:42:30.921928 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:42:30.922396 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:42:30.922400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:42:30.922402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:42:30.922489 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:42:30.922791 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:42:30.922795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:42:30.922797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:42:30.922995 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:42:30.923102 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:30.923461 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 14:42:30.923578 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:42:30.927361 13074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:42:30.931874 13074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:42:30.952306 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.943384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.1.201.3:32955)
0928 14:42:30.944096 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:30.944119 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:30.945084 (+ 965us) server_negotiation.cc:407] Connection header received
0928 14:42:30.945177 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:30.945182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:30.945397 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:30.945553 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:42:30.946818 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.947973 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.950503 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.951316 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.951397 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.951603 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:30.952029 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 14:42:30.952144 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":112,"threads_started":1}
I20260928 14:42:30.952925 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.943002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:32955 (local address 127.0.0.1:37126)
0928 14:42:30.944962 (+ 1960us) negotiation.cc:107] Waiting for socket to connect
0928 14:42:30.944973 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:42:30.945061 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:42:30.945774 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:42:30.945809 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:42:30.945831 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:42:30.945974 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.945982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.948130 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.948133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.950343 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.950354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.951699 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.951705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.951824 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.951862 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:42:30.952520 (+ 658us) client_negotiation.cc:240] Negotiation successful
0928 14:42:30.952755 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1912,"mutex_wait_us":24,"thread_start_us":90,"threads_started":1}
I20260928 14:42:30.953212 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.942788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39083 (local address 127.0.0.1:39334)
0928 14:42:30.944722 (+ 1934us) negotiation.cc:107] Waiting for socket to connect
0928 14:42:30.944733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:42:30.944794 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:42:30.945762 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:42:30.945769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:42:30.945817 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:42:30.945969 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.945977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.948283 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.948286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.949562 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.949573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.952388 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.952393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.952773 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.952793 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:42:30.952838 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 14:42:30.952929 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1881,"mutex_wait_us":4,"thread_start_us":88,"threads_started":1}
I20260928 14:42:30.956902 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.943197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.1.201.2:39083)
0928 14:42:30.945287 (+ 2090us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:30.945302 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:30.945323 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:42:30.945394 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:30.945399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:30.945588 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:30.946135 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:42:30.946266 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.947473 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.949736 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.950593 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.953714 (+ 3121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.956568 (+ 2854us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:30.956627 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 14:42:30.956731 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1991,"thread_start_us":108,"threads_started":1}
I20260928 14:42:30.957057 13473 tablet_service.cc:1467] Processing CreateTablet for tablet 6912673cea204d589d6db7bdc46fad1c (DEFAULT_TABLE table=test-table [id=8de0c79dad55433fa56abeb32bb4288e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:42:30.958303 13473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6912673cea204d589d6db7bdc46fad1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:42:30.959960 13357 tablet_service.cc:1467] Processing CreateTablet for tablet 6912673cea204d589d6db7bdc46fad1c (DEFAULT_TABLE table=test-table [id=8de0c79dad55433fa56abeb32bb4288e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:42:30.960999 13357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6912673cea204d589d6db7bdc46fad1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:42:30.963783 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.941608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42705 (local address 127.0.0.1:54896)
0928 14:42:30.944356 (+ 2748us) negotiation.cc:107] Waiting for socket to connect
0928 14:42:30.944405 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:42:30.944526 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:42:30.950737 (+ 6211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:42:30.950741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:42:30.950760 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:42:30.950858 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.950863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.954189 (+ 3326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.954193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.955255 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.955265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.963334 (+ 8069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.963339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.963457 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.963490 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:42:30.963551 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 14:42:30.963654 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2576,"thread_start_us":87,"threads_started":1}
I20260928 14:42:30.964708 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.941831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.1.201.1:42705)
0928 14:42:30.947729 (+ 5898us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:30.947736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:30.947758 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:42:30.947912 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:30.947917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:30.948118 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:30.951004 (+ 2886us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:42:30.951161 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.952229 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.955905 (+ 3676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.956687 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.964218 (+ 7531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:30.964395 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:30.964450 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 14:42:30.964557 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5791,"thread_start_us":114,"threads_started":1}
I20260928 14:42:30.967649 13210 tablet_service.cc:1467] Processing CreateTablet for tablet 6912673cea204d589d6db7bdc46fad1c (DEFAULT_TABLE table=test-table [id=8de0c79dad55433fa56abeb32bb4288e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:42:30.968657 13210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6912673cea204d589d6db7bdc46fad1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:42:30.970023 13577 tablet_bootstrap.cc:492] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a: Bootstrap starting.
I20260928 14:42:30.970952 13576 tablet_bootstrap.cc:492] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: Bootstrap starting.
I20260928 14:42:30.973030 13577 tablet_bootstrap.cc:654] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a: Neither blocks nor log segments found. Creating new log.
I20260928 14:42:30.973614 13576 tablet_bootstrap.cc:654] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: Neither blocks nor log segments found. Creating new log.
I20260928 14:42:30.973970 13577 log.cc:824] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:42:30.974547 13576 log.cc:824] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: Log is configured to *not* fsync() on all Append() calls
I20260928 14:42:30.975785 13580 tablet_bootstrap.cc:492] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615: Bootstrap starting.
I20260928 14:42:30.976786 13576 tablet_bootstrap.cc:492] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: No bootstrap required, opened a new log
I20260928 14:42:30.976979 13576 ts_tablet_manager.cc:1397] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:42:30.978233 13580 tablet_bootstrap.cc:654] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615: Neither blocks nor log segments found. Creating new log.
I20260928 14:42:30.978999 13580 log.cc:824] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615: Log is configured to *not* fsync() on all Append() calls
I20260928 14:42:30.980885 13580 tablet_bootstrap.cc:492] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615: No bootstrap required, opened a new log
I20260928 14:42:30.981050 13580 ts_tablet_manager.cc:1397] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:42:30.980867 13576 raft_consensus.cc:348] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 } } peers { permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 } } peers { permanent_uuid: "9e40636a5782448287f18e5085f958ad" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 32955 } }
I20260928 14:42:30.981132 13576 raft_consensus.cc:374] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:42:30.981179 13576 raft_consensus.cc:729] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e40636a5782448287f18e5085f958ad, State: Initialized, Role: FOLLOWER
I20260928 14:42:30.981627 13576 consensus_queue.cc:260] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 } } peers { permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 } } peers { permanent_uuid: "9e40636a5782448287f18e5085f958ad" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 32955 } }
I20260928 14:42:30.982596 13576 ts_tablet_manager.cc:1428] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:42:30.983891 13558 heartbeater.cc:503] Master 127.1.201.62:35787 was elected leader, sending a full tablet report...
I20260928 14:42:30.987550 13577 tablet_bootstrap.cc:492] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a: No bootstrap required, opened a new log
I20260928 14:42:30.987771 13577 ts_tablet_manager.cc:1397] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260928 14:42:30.988885 13580 raft_consensus.cc:348] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 } } peers { permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 } } peers { permanent_uuid: "9e40636a5782448287f18e5085f958ad" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 32955 } }
I20260928 14:42:30.989166 13580 raft_consensus.cc:374] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:42:30.989225 13580 raft_consensus.cc:729] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90194cc139d14f4abccfc385117df615, State: Initialized, Role: FOLLOWER
I20260928 14:42:30.989734 13580 consensus_queue.cc:260] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 } } peers { permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 } } peers { permanent_uuid: "9e40636a5782448287f18e5085f958ad" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 32955 } }
I20260928 14:42:30.990820 13580 ts_tablet_manager.cc:1428] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260928 14:42:30.991891 13288 heartbeater.cc:503] Master 127.1.201.62:35787 was elected leader, sending a full tablet report...
I20260928 14:42:30.991817 13577 raft_consensus.cc:348] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 } } peers { permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 } } peers { permanent_uuid: "9e40636a5782448287f18e5085f958ad" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 32955 } }
I20260928 14:42:30.992061 13577 raft_consensus.cc:374] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:42:30.992108 13577 raft_consensus.cc:729] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ddd0e0ed694d599a1f3fa09f88b38a, State: Initialized, Role: FOLLOWER
I20260928 14:42:30.992508 13577 consensus_queue.cc:260] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 } } peers { permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 } } peers { permanent_uuid: "9e40636a5782448287f18e5085f958ad" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 32955 } }
I20260928 14:42:30.993448 13577 ts_tablet_manager.cc:1428] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:42:30.993795 13424 heartbeater.cc:503] Master 127.1.201.62:35787 was elected leader, sending a full tablet report...
I20260928 14:42:30.994699 13582 raft_consensus.cc:482] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:42:30.994870 13582 raft_consensus.cc:504] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 } } peers { permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 } } peers { permanent_uuid: "9e40636a5782448287f18e5085f958ad" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 32955 } }
W20260928 14:42:30.995904 13425 tablet.cc:2369] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:42:30.996040 13582 leader_election.cc:288] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90194cc139d14f4abccfc385117df615 (127.1.201.1:42705), f7ddd0e0ed694d599a1f3fa09f88b38a (127.1.201.2:39083)
I20260928 14:42:31.002792 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.996928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:34891 (local address 127.1.201.1:42705)
0928 14:42:30.997037 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:30.997041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:30.998567 (+ 1526us) server_negotiation.cc:407] Connection header received
0928 14:42:30.998878 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:30.998881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:30.998920 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:30.998962 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:42:30.999306 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:31.000038 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:31.001433 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:31.002077 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:31.002159 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:31.002219 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:31.002573 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 14:42:31.002663 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 14:42:31.003425 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.996277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39083 (local address 127.1.201.3:36477)
0928 14:42:30.996511 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 14:42:30.996526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:42:30.996626 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:42:30.997074 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:42:30.997079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:42:30.997106 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:42:30.997231 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.997239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.998256 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.998260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:30.999438 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.999447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:31.003004 (+ 3557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:31.003008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:31.003119 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:31.003143 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:42:31.003221 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:42:31.003297 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260928 14:42:31.003418 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.996581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42705 (local address 127.1.201.3:34891)
0928 14:42:30.998490 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0928 14:42:30.998504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:42:30.998758 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:42:30.999078 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:42:30.999081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:42:30.999097 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:42:30.999198 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:30.999203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:31.000173 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:31.000176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:31.001288 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:42:31.001300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:42:31.002319 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:42:31.002322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:42:31.002424 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:31.002446 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:42:31.003166 (+ 720us) client_negotiation.cc:240] Negotiation successful
0928 14:42:31.003249 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1848,"thread_start_us":110,"threads_started":1}
I20260928 14:42:31.004528 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6912673cea204d589d6db7bdc46fad1c" candidate_uuid: "9e40636a5782448287f18e5085f958ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90194cc139d14f4abccfc385117df615" is_pre_election: true
I20260928 14:42:31.004880 13233 raft_consensus.cc:2454] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e40636a5782448287f18e5085f958ad in term 0.
I20260928 14:42:31.005976 13446 leader_election.cc:302] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90194cc139d14f4abccfc385117df615, 9e40636a5782448287f18e5085f958ad; no voters:
I20260928 14:42:31.006788 13584 raft_consensus.cc:2781] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:42:31.006955 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:30.996767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:36477 (local address 127.1.201.2:39083)
0928 14:42:30.996876 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:30.996880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:30.996892 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:42:30.996934 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:30.996937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:30.996978 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:30.997053 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:42:30.997369 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:30.998111 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:30.999595 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:31.000207 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:31.000304 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:31.000368 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:31.006731 (+ 6363us) server_negotiation.cc:299] Negotiation successful
0928 14:42:31.006828 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:42:31.006942 13584 raft_consensus.cc:482] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:42:31.007153 13584 raft_consensus.cc:3037] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:42:31.007491 13378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6912673cea204d589d6db7bdc46fad1c" candidate_uuid: "9e40636a5782448287f18e5085f958ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" is_pre_election: true
I20260928 14:42:31.007774 13378 raft_consensus.cc:2454] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e40636a5782448287f18e5085f958ad in term 0.
I20260928 14:42:31.009806 13584 raft_consensus.cc:504] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 } } peers { permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 } } peers { permanent_uuid: "9e40636a5782448287f18e5085f958ad" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 32955 } }
I20260928 14:42:31.010550 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6912673cea204d589d6db7bdc46fad1c" candidate_uuid: "9e40636a5782448287f18e5085f958ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90194cc139d14f4abccfc385117df615"
I20260928 14:42:31.010691 13233 raft_consensus.cc:3037] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:42:31.010951 13378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6912673cea204d589d6db7bdc46fad1c" candidate_uuid: "9e40636a5782448287f18e5085f958ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a"
I20260928 14:42:31.011073 13378 raft_consensus.cc:3037] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:42:31.012068 13233 raft_consensus.cc:2454] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e40636a5782448287f18e5085f958ad in term 1.
I20260928 14:42:31.012347 13378 raft_consensus.cc:2454] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e40636a5782448287f18e5085f958ad in term 1.
I20260928 14:42:31.012461 13446 leader_election.cc:302] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90194cc139d14f4abccfc385117df615, 9e40636a5782448287f18e5085f958ad; no voters:
I20260928 14:42:31.012637 13582 raft_consensus.cc:2781] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:42:31.012830 13584 leader_election.cc:288] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [CANDIDATE]: Term 1 election: Requested vote from peers 90194cc139d14f4abccfc385117df615 (127.1.201.1:42705), f7ddd0e0ed694d599a1f3fa09f88b38a (127.1.201.2:39083)
I20260928 14:42:31.012861 13582 raft_consensus.cc:686] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 1 LEADER]: Becoming Leader. State: Replica: 9e40636a5782448287f18e5085f958ad, State: Running, Role: LEADER
I20260928 14:42:31.013253 13582 consensus_queue.cc:237] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 } } peers { permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 } } peers { permanent_uuid: "9e40636a5782448287f18e5085f958ad" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 32955 } }
I20260928 14:42:31.016798 13074 catalog_manager.cc:5495] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e40636a5782448287f18e5085f958ad (127.1.201.3). New cstate: current_term: 1 leader_uuid: "9e40636a5782448287f18e5085f958ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e40636a5782448287f18e5085f958ad" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 32955 } health_report { overall_health: HEALTHY } } }
I20260928 14:42:31.035374 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:31.030642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.1.201.3:32955)
0928 14:42:31.030759 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:42:31.030764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:42:31.030779 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:42:31.030860 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:42:31.030863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:42:31.030924 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:42:31.031035 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:42:31.031284 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:31.032077 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:31.033383 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:42:31.033706 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:42:31.033857 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:42:31.034298 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:42:31.034889 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:42:31.035014 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 14:42:31.035166 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 14:42:31.035223 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:42:31.042927 13378 raft_consensus.cc:1261] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a [term 1 FOLLOWER]: Refusing update from remote peer 9e40636a5782448287f18e5085f958ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:42:31.043675 13584 consensus_queue.cc:1035] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ddd0e0ed694d599a1f3fa09f88b38a" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 28 14:42:31 dist-test-slave-0n27 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:42:31.045136 13233 raft_consensus.cc:1261] T 6912673cea204d589d6db7bdc46fad1c P 90194cc139d14f4abccfc385117df615 [term 1 FOLLOWER]: Refusing update from remote peer 9e40636a5782448287f18e5085f958ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:42:31.045701 13584 consensus_queue.cc:1035] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "90194cc139d14f4abccfc385117df615" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:42:31.060444 13592 mvcc.cc:205] Tried to move back new op lower bound from 7334324433053544448 to 7334324432943747072. Current Snapshot: MvccSnapshot[applied={T|T < 7334324433053544448}]
I20260928 14:42:31.060719 13591 mvcc.cc:205] Tried to move back new op lower bound from 7334324433053544448 to 7334324432943747072. Current Snapshot: MvccSnapshot[applied={T|T < 7334324433053544448 or (T in {7334324433053544448})}]
I20260928 14:42:31.070017 13072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:42:31.070200 13072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:42:31.072741 13072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bd03dea082244b580927d6f5a927928: Sending DeleteTablet for 3 replicas of tablet 6912673cea204d589d6db7bdc46fad1c
I20260928 14:42:31.073539 13473 tablet_service.cc:1514] Processing DeleteTablet for tablet 6912673cea204d589d6db7bdc46fad1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:42:31 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:37126
I20260928 14:42:31.073673 13357 tablet_service.cc:1514] Processing DeleteTablet for tablet 6912673cea204d589d6db7bdc46fad1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:42:31 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:39334
I20260928 14:42:31.074328 13210 tablet_service.cc:1514] Processing DeleteTablet for tablet 6912673cea204d589d6db7bdc46fad1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:42:31 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:54896
I20260928 14:42:31.075006 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 13155
I20260928 14:42:31.084046 13600 tablet_replica.cc:330] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a: stopping tablet replica
I20260928 14:42:31.084414 13600 raft_consensus.cc:2229] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:42:31.084816 13600 raft_consensus.cc:2258] T 6912673cea204d589d6db7bdc46fad1c P f7ddd0e0ed694d599a1f3fa09f88b38a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:42:31.085264 13599 tablet_replica.cc:330] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: stopping tablet replica
I20260928 14:42:31.085613 13599 raft_consensus.cc:2229] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:42:31.086030 13599 raft_consensus.cc:2258] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:42:31.087543 13599 ts_tablet_manager.cc:1905] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:42:31.094467 13599 ts_tablet_manager.cc:1918] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:42:31.095625 13599 log.cc:1196] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/6912673cea204d589d6db7bdc46fad1c
I20260928 14:42:31.096086 13599 ts_tablet_manager.cc:1939] T 6912673cea204d589d6db7bdc46fad1c P 9e40636a5782448287f18e5085f958ad: Deleting consensus metadata
I20260928 14:42:31.097395 13059 catalog_manager.cc:4841] TS 9e40636a5782448287f18e5085f958ad (127.1.201.3:32955): tablet 6912673cea204d589d6db7bdc46fad1c (table test-table [id=8de0c79dad55433fa56abeb32bb4288e]) successfully deleted
W20260928 14:42:31.098788 13059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:42705 (error 108)
I20260928 14:42:31.099709 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 13294
I20260928 14:42:31.106086 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:31.101000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42705 (local address 127.0.0.1:54904)
0928 14:42:31.105903 (+ 4903us) negotiation.cc:107] Waiting for socket to connect
0928 14:42:31.105983 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:42705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4807}
W20260928 14:42:31.106300 13059 catalog_manager.cc:4584] TS 90194cc139d14f4abccfc385117df615 (127.1.201.1:42705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6912673cea204d589d6db7bdc46fad1c: Network error: Client connection negotiation failed: client connection to 127.1.201.1:42705: connect: Connection refused (error 111)
I20260928 14:42:31.108753 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 13429
I20260928 14:42:31.109755 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:42:31.108952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39083 (local address 127.0.0.1:39350)
0928 14:42:31.109608 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 14:42:31.109670 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.2:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564}
I20260928 14:42:31.117023 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 13042
I20260928 14:42:31.125188 1828 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:42:31.469887 1828 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:42:31.470031 1828 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:42:31.814863 1828 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:42:31.815 UTC [12628] LOG: received smart shutdown request
2026-09-28 14:42:31.818 UTC [12628] LOG: background worker "logical replication launcher" (PID 12637) exited with exit code 1
2026-09-28 14:42:31.818 UTC [12632] LOG: shutting down
2026-09-28 14:42:31.967 UTC [12628] LOG: database system is shut down
2026-09-28T14:42:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45173 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:42:32 dist-test-slave-0n27 krb5kdc[13613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:42:32 dist-test-slave-0n27 krb5kdc[13613](info): set up 2 sockets
Sep 28 14:42:32 dist-test-slave-0n27 krb5kdc[13613](info): commencing operation
krb5kdc: starting...
W20260928 14:42:33.114852 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:42:33 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:42:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.201.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.201.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/rangeradmin_127.1.201.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/rangeradmin_127.1.201.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.201.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.201.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/rangerlookup_127.1.201.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/rangerlookup_127.1.201.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.201.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.201.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/HTTP_127.1.201.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/HTTP_127.1.201.20@KRBTEST.COM.keytab.
I20260928 14:42:33.279937 1828 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskScXicz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/postgres -l logfile start
2026-09-28 14:42:35.584 UTC [13643] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:42:35.585 UTC [13643] LOG: listening on IPv4 address "127.1.201.20", port 55779
2026-09-28 14:42:35.588 UTC [13643] LOG: listening on Unix socket "/tmp/.s.PGSQL.55779"
2026-09-28 14:42:35.622 UTC [13646] LOG: database system was shut down at 2026-09-28 14:42:34 UTC
2026-09-28 14:42:35.626 UTC [13643] LOG: database system is ready to accept connections
I20260928 14:42:36.542893 1828 mini_postgres.cc:96] Postgres bound to 55779
2026-09-28 14:42:36.549 UTC [13654] FATAL: database "slave" does not exist
127.1.201.20:55779 - accepting connections
I20260928 14:42:36.549649 1828 mini_ranger.cc:161] Starting Ranger...
I20260928 14:42:36.564051 1828 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:42:36.969810 1828 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:42:36.969904 1828 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-admin
2026-09-28 14:42:37,224 [I] DB FLAVOR :POSTGRES
2026-09-28 14:42:37,224 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:42:37,224 [I] Checking connection..
2026-09-28 14:42:37,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:42:37,472 [I] Checking connection passed.
2026-09-28 14:42:37,472 [I] --------- Verifying version history table ---------
2026-09-28 14:42:37,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:42:37,716 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:42:37,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:42:37,973 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:42:37,973 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:42:37,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:42:38.189 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 14:42:38,196 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:42:38,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:42:38,438 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:42:38,439 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:42:38,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:42:38,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:42:38,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0n27.c.gcp-upstream.internal','N') ;"
2026-09-28 14:42:39,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:42:39,435 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:42:39,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:42:39,681 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:42:39,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:42:39,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:42:40,192 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:42:40,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:42:41.102 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.118 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.134 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.148 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.163 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.179 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.233 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.240 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.253 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.263 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.272 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.285 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.294 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.300 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.306 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41.312 UTC [13826] WARNING: there is no transaction in progress
2026-09-28 14:42:41,886 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:42:41,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:42:42,114 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:42:42,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:42:42,366 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:42:42,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:42:42,611 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:42:42,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:42:42,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:42:43,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0n27.c.gcp-upstream.internal';"
2026-09-28 14:42:43,343 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:42:43,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:42:43,607 [I] DB_PATCHES have already been applied
I20260928 14:42:43.613031 1828 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-admin:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskScXicz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:42:43.613129 1828 mini_ranger.cc:193] Using host: 127.1.201.20
I20260928 14:42:43.616844 1828 mini_ranger.cc:227] Ranger admin URL: http://127.1.201.20:43731
Sep 28, 2026 2:42:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:42:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:42:44.725178 1828 mini_ranger.cc:160] Time spent starting Ranger: real 8.176s user 0.003s sys 0.007s
Sep 28, 2026 2:43:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:43:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:43:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:43:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:43:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:43:06.213032 1828 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.201.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.201.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/rangerkms_127.1.201.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.201.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/rangerkms_127.1.201.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.201.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.201.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.201.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/HTTP_127.1.201.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.201.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/HTTP_127.1.201.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:43:06 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606586, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:43:06.293740 1828 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:43:06.303208 1828 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:43:07.123638 1828 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:43:07.123711 1828 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-kms
I20260928 14:43:07.123724 1828 mini_ranger_kms.cc:227] Using postgres at 127.1.201.20:55779
2026-09-28 14:43:07,291 [I] DB FLAVOR :POSTGRES
2026-09-28 14:43:07,291 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:43:07,291 [I] Checking connection
2026-09-28 14:43:07,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:43:07,543 [I] connection success
2026-09-28 14:43:07,543 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:43:07,543 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:43:07,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:43:07,785 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:43:07,785 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:43:07,785 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:43:07,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.201.20:55779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:43:08,057 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:43:08.518623 1828 mini_ranger_kms.cc:310] Created kms service
I20260928 14:43:08.717029 1828 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:43:08.763141 1828 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:43:08 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606586, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:09.132905 1828 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:43:09.133047 1828 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-kms:/tmp/dist-test-taskScXicz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskScXicz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:43:09.133066 1828 mini_ranger_kms.cc:240] Using host: 127.1.201.20
I20260928 14:43:09.139034 1828 mini_ranger_kms.cc:276] Ranger KMS PID: 14006
I20260928 14:43:09.139117 1828 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.201.20:47503
Sep 28, 2026 2:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:43:11.327431 1828 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.034s user 0.017s sys 0.136s
I20260928 14:43:11.327546 1828 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:43:11.327623 1828 mini_ranger_kms.cc:401] 127.1.201.20:47503/kms/v1/keys
Sep 28 14:43:11 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606591, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.201.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:43:11 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606591, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.201.20@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
Sep 28 14:43:12 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606591, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.201.20@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
Sep 28 14:43:14 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:43:14.360280 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:39161
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:39161
--ranger_config_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:47503/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:14.470067 14057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:14.470335 14057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:14.470369 14057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:14.474109 14057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:43:14.474156 14057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:14.474175 14057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:14.474190 14057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:43:14.474205 14057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:43:14.479341 14057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.201.20:47503/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:39161
--ranger_config_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:14.480379 14057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:14.481374 14057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:43:14.490096 14057 server_base.cc:1034] running on GCE node
W20260928 14:43:14.490168 14062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:14.490294 14065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:14.490367 14063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:14.491185 14057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:14.492514 14057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:14.493709 14057 hybrid_clock.cc:648] HybridClock initialized: now 1790606594493681 us; error 42 us; skew 500 ppm
Sep 28 14:43:14 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606594, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:14.503327 14057 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:43:14.504349 14057 webserver.cc:460] Webserver started at http://127.1.201.62:40759/ using document root <none> and password file <none>
I20260928 14:43:14.504897 14057 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:14.504951 14057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:14.505167 14057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:43:14 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606594, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:14.564797 14057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "2904a37fde2249db9fc69f70cb260894"
format_stamp: "Formatted at 2026-09-28 14:43:14 on dist-test-slave-0n27"
server_key: "d7be8c1472d848bca284ad1c939de2fe"
server_key_iv: "137220cc2ad4605eea4d80ed75fad61c"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:14.565418 14057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "2904a37fde2249db9fc69f70cb260894"
format_stamp: "Formatted at 2026-09-28 14:43:14 on dist-test-slave-0n27"
server_key: "d7be8c1472d848bca284ad1c939de2fe"
server_key_iv: "137220cc2ad4605eea4d80ed75fad61c"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:14.569615 14057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:43:14.587604 14072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:14.589111 14057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:43:14.589254 14057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "2904a37fde2249db9fc69f70cb260894"
format_stamp: "Formatted at 2026-09-28 14:43:14 on dist-test-slave-0n27"
server_key: "d7be8c1472d848bca284ad1c939de2fe"
server_key_iv: "137220cc2ad4605eea4d80ed75fad61c"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:14.589391 14057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:14.601126 14057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:14.603904 14057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:14.604090 14057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:14.612219 14057 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:39161
I20260928 14:43:14.612224 14124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:39161 every 8 connection(s)
I20260928 14:43:14.613178 14057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:43:14.615973 14125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:14.617692 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 14057
I20260928 14:43:14.617854 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:43:14.618172 1828 external_mini_cluster.cc:1428] Setting key fd94a63e58f2629688ae8736b9b7c8d4
I20260928 14:43:14.622754 14125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: Bootstrap starting.
I20260928 14:43:14.625135 14125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:14.626040 14125 log.cc:824] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:14.628046 14125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: No bootstrap required, opened a new log
Sep 28 14:43:14 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:14.632227 14125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } }
I20260928 14:43:14.632485 14125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:14.632556 14125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2904a37fde2249db9fc69f70cb260894, State: Initialized, Role: FOLLOWER
I20260928 14:43:14.633149 14125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } }
I20260928 14:43:14.633529 14125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:43:14.633628 14125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:43:14.633747 14125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:14.634354 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:14.619656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.1.201.62:39161)
0928 14:43:14.620092 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:14.620112 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:14.620154 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:43:14.620963 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:14.620989 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:14.621390 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:14.621754 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:14.622585 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:14.624005 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:14.625293 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:14.625993 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:14.626152 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:14.629186 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:14.629193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:14.629211 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:14.629241 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:14.631235 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:14.631738 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:14.631750 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:14.631756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:14.631878 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:14.632178 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:14.632181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:14.632183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:14.632675 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:14.632894 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:14.633163 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 14:43:14.633420 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":115,"threads_started":1}
I20260928 14:43:14.634739 14125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } }
I20260928 14:43:14.635073 14125 leader_election.cc:302] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2904a37fde2249db9fc69f70cb260894; no voters:
I20260928 14:43:14.635412 14125 leader_election.cc:288] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:43:14.635787 14131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:43:14.636005 14131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 1 LEADER]: Becoming Leader. State: Replica: 2904a37fde2249db9fc69f70cb260894, State: Running, Role: LEADER
I20260928 14:43:14.636632 14125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:43:14.636323 14131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } }
I20260928 14:43:14.637857 14131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2904a37fde2249db9fc69f70cb260894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } } }
I20260928 14:43:14.638041 14131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:14.638294 14131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2904a37fde2249db9fc69f70cb260894. Latest consensus state: current_term: 1 leader_uuid: "2904a37fde2249db9fc69f70cb260894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } } }
I20260928 14:43:14.638427 14131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:14.638844 14125 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:43:14.642699 14140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:43:14.646324 14140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:43:14.652837 14140 catalog_manager.cc:1347] Generated new cluster ID: 84e17fc89e8f4603a42cf771ca885ea7
I20260928 14:43:14.652892 14140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:43:14.666235 14140 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:43:14.667578 14140 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:43:14.673918 14140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: Generated new TSK 0
I20260928 14:43:14.674711 14140 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:43:15.340698 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:47503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39161
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:15.480911 14170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:15.481149 14170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:15.481190 14170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:15.486264 14170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:15.486402 14170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:15.486536 14170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:43:15.493181 14170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.201.20:47503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:15.494495 14170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:15.495500 14170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:15.504930 14178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:15.505273 14170 server_base.cc:1034] running on GCE node
W20260928 14:43:15.505362 14180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:15.505033 14177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:15.505946 14170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:15.506646 14170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:15.507925 14170 hybrid_clock.cc:648] HybridClock initialized: now 1790606595507881 us; error 57 us; skew 500 ppm
Sep 28 14:43:15 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:15.510946 14170 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:43:15.512022 14170 webserver.cc:460] Webserver started at http://127.1.201.1:46053/ using document root <none> and password file <none>
I20260928 14:43:15.512741 14170 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:15.512840 14170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:15.513093 14170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:43:15 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:15.532630 14170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "40f4090fbf0441a69ff455eefc90ced6"
format_stamp: "Formatted at 2026-09-28 14:43:15 on dist-test-slave-0n27"
server_key: "fb92333d310f4d80552707e4ba55ca72"
server_key_iv: "8024395e1e0cbc8703b70b9ce5fd9799"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:15.533378 14170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "40f4090fbf0441a69ff455eefc90ced6"
format_stamp: "Formatted at 2026-09-28 14:43:15 on dist-test-slave-0n27"
server_key: "fb92333d310f4d80552707e4ba55ca72"
server_key_iv: "8024395e1e0cbc8703b70b9ce5fd9799"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:15.538632 14170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 14:43:15.558169 14187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:15.559825 14170 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.002s
I20260928 14:43:15.559981 14170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "40f4090fbf0441a69ff455eefc90ced6"
format_stamp: "Formatted at 2026-09-28 14:43:15 on dist-test-slave-0n27"
server_key: "fb92333d310f4d80552707e4ba55ca72"
server_key_iv: "8024395e1e0cbc8703b70b9ce5fd9799"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:15.560137 14170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:15.595320 14170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:15.599280 14170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:15.599520 14170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:15.600257 14170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:15.601392 14170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:15.601513 14170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:15.601641 14170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:15.601724 14170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:15.612973 14170 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:39851
I20260928 14:43:15.614141 14170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:43:15.618146 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 14170
I20260928 14:43:15.618255 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:43:15.618525 1828 external_mini_cluster.cc:1428] Setting key d1b819171b2567aa7f0d2dce907fe058
I20260928 14:43:15.630157 14301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:39851 every 8 connection(s)
Sep 28 14:43:15 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:15.643970 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:15.617630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39161 (local address 127.1.201.1:35389)
0928 14:43:15.622071 (+ 4441us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:15.622139 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:15.623150 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:15.631514 (+ 8364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:15.631525 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:15.631884 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:15.632607 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:15.632629 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:15.633827 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:15.633831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:15.635238 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:15.635249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:15.635930 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:15.635935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:15.636126 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:15.636833 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:15.636853 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:15.638398 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:15.640824 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:15.640831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:15.640847 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:15.641170 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:15.641477 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:15.641480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:15.641482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:15.641657 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:15.642129 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:15.642136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:15.642450 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 14:43:15.642647 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 14:43:15.642966 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4209,"thread_start_us":155,"threads_started":1}
I20260928 14:43:15.645030 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:15.617127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:35389 (local address 127.1.201.62:39161)
0928 14:43:15.631071 (+ 13944us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:15.631080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:15.631108 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:43:15.631152 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:15.631154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:15.631200 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:15.631296 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:15.632770 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:15.633639 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:15.635386 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:15.635751 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:15.637524 (+ 1773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:15.638567 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:15.638571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:15.638574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:15.638602 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:15.640675 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:15.641286 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:15.641289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:15.641291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:15.641378 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:15.641767 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:15.641770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:15.641772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:15.642006 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:15.644654 (+ 2648us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:15.644747 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 14:43:15.644864 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13854,"thread_start_us":105,"threads_started":1}
I20260928 14:43:15.646369 14303 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39161
I20260928 14:43:15.646787 14303 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:15.647370 14303 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
I20260928 14:43:15.649215 14089 ts_manager.cc:194] Registered new tserver with Master: 40f4090fbf0441a69ff455eefc90ced6 (127.1.201.1:39851)
I20260928 14:43:15.650804 14089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:35389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:43:15.680632 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:47503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39161
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:43:15 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 14:43:15.835567 14309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:15.836019 14309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:15.836131 14309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:15.845049 14309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:15.845129 14309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:15.845201 14309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:43:15.852542 14309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.201.20:47503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:15.854080 14309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:15.855432 14309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:43:15.867888 14309 server_base.cc:1034] running on GCE node
W20260928 14:43:15.870615 14316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:15.870929 14318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:15.873924 14315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:15.874428 14309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:15.875203 14309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:15.882568 14309 hybrid_clock.cc:648] HybridClock initialized: now 1790606595882452 us; error 140 us; skew 500 ppm
Sep 28 14:43:15 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:15.888306 14309 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:43:15.890234 14309 webserver.cc:460] Webserver started at http://127.1.201.2:42673/ using document root <none> and password file <none>
I20260928 14:43:15.890965 14309 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:15.891093 14309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:15.891376 14309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:43:15 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:15.911922 14309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "697d875a79c04d1fa263560555eb7eb4"
format_stamp: "Formatted at 2026-09-28 14:43:15 on dist-test-slave-0n27"
server_key: "5cd76df6502d39feaeb099ef781d2a79"
server_key_iv: "63a9496ff66d8c90788a5c9a2744df7d"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:15.912737 14309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "697d875a79c04d1fa263560555eb7eb4"
format_stamp: "Formatted at 2026-09-28 14:43:15 on dist-test-slave-0n27"
server_key: "5cd76df6502d39feaeb099ef781d2a79"
server_key_iv: "63a9496ff66d8c90788a5c9a2744df7d"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:15.917930 14309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:43:15.942220 14325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:15.943943 14309 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260928 14:43:15.944067 14309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "697d875a79c04d1fa263560555eb7eb4"
format_stamp: "Formatted at 2026-09-28 14:43:15 on dist-test-slave-0n27"
server_key: "5cd76df6502d39feaeb099ef781d2a79"
server_key_iv: "63a9496ff66d8c90788a5c9a2744df7d"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:15.944204 14309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:15.976991 14309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:15.980839 14309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:15.981087 14309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:15.981986 14309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:15.983119 14309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:15.983239 14309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:15.983345 14309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:15.983412 14309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:15.997660 14309 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:38363
I20260928 14:43:15.998838 14309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:43:16.000102 14438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:38363 every 8 connection(s)
I20260928 14:43:16.001670 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 14309
I20260928 14:43:16.001816 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:43:16.002168 1828 external_mini_cluster.cc:1428] Setting key 76fd47dc7a0713d4849ab3c552370053
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
Sep 28 14:43:16 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
I20260928 14:43:16.028414 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:16.005660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:56933 (local address 127.1.201.62:39161)
0928 14:43:16.005825 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:16.005831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:16.014173 (+ 8342us) server_negotiation.cc:407] Connection header received
0928 14:43:16.015768 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:16.015772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:16.015839 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:16.015899 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:16.017686 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.018615 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.020580 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.020940 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.021022 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:16.024146 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:16.024151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:16.024154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:16.024184 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:16.025992 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:16.026598 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:16.026602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:16.026604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:16.026689 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:16.027108 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:16.027112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:16.027113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:16.027328 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:16.027394 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:16.028114 (+ 720us) server_negotiation.cc:299] Negotiation successful
0928 14:43:16.028246 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:43:16.029392 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:16.004582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39161 (local address 127.1.201.2:56933)
0928 14:43:16.013947 (+ 9365us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:16.014030 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:16.015459 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:16.016375 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:16.016386 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:16.016766 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:16.017502 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:16.017522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.018923 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.018926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:16.020430 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:16.020442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.021245 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.021249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:16.021449 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:16.022151 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:16.022173 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:16.023931 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:16.026133 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:16.026140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:16.026155 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:16.026476 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:16.026796 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:16.026799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:16.026801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:16.026990 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:16.027473 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:16.027479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:16.027797 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 14:43:16.028028 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 14:43:16.028366 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9017,"thread_start_us":125,"threads_started":1}
I20260928 14:43:16.030725 14439 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39161
I20260928 14:43:16.031001 14439 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:16.031507 14439 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
I20260928 14:43:16.032579 14089 ts_manager.cc:194] Registered new tserver with Master: 697d875a79c04d1fa263560555eb7eb4 (127.1.201.2:38363)
I20260928 14:43:16.033293 14089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:56933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:43:16.044158 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:47503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39161
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:16.179546 14444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:16.179890 14444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:16.180003 14444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:16.185832 14444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:16.185917 14444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:16.186017 14444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:43:16.191895 14444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.201.20:47503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:16.193284 14444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:16.194733 14444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:16.207535 14452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:16.208055 14450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:16.208894 14444 server_base.cc:1034] running on GCE node
W20260928 14:43:16.210218 14449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:16.210793 14444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:16.211583 14444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 28 14:43:16 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:16.214869 14444 hybrid_clock.cc:648] HybridClock initialized: now 1790606596214791 us; error 80 us; skew 500 ppm
Sep 28 14:43:16 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606596, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:16.218688 14444 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:43:16.219794 14444 webserver.cc:460] Webserver started at http://127.1.201.3:38629/ using document root <none> and password file <none>
I20260928 14:43:16.220588 14444 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:16.220741 14444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:16.221043 14444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:43:16 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606596, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:16.238235 14444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "bbee1c7f1eea420e8b1b235a1c9781e0"
format_stamp: "Formatted at 2026-09-28 14:43:16 on dist-test-slave-0n27"
server_key: "2f99d8333e3f6b5f02328442e745565b"
server_key_iv: "a943870873411fb2b35dbc5845ef347d"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:16.239069 14444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "bbee1c7f1eea420e8b1b235a1c9781e0"
format_stamp: "Formatted at 2026-09-28 14:43:16 on dist-test-slave-0n27"
server_key: "2f99d8333e3f6b5f02328442e745565b"
server_key_iv: "a943870873411fb2b35dbc5845ef347d"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:16.250558 14444 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.000s sys 0.003s
I20260928 14:43:16.275126 14459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:16.284648 14444 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.000s
I20260928 14:43:16.284897 14444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "bbee1c7f1eea420e8b1b235a1c9781e0"
format_stamp: "Formatted at 2026-09-28 14:43:16 on dist-test-slave-0n27"
server_key: "2f99d8333e3f6b5f02328442e745565b"
server_key_iv: "a943870873411fb2b35dbc5845ef347d"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:16.285142 14444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:16.308575 14444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:16.312225 14444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:16.312455 14444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:16.313184 14444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:16.314388 14444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:16.314491 14444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:16.314587 14444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:16.314667 14444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:16.324806 14444 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:34219
I20260928 14:43:16.326095 14444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:43:16.330178 14572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:34219 every 8 connection(s)
I20260928 14:43:16.333854 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 14444
I20260928 14:43:16.333992 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:43:16.334321 1828 external_mini_cluster.cc:1428] Setting key 05b3f219141541752818ae68cd6f7c71
Sep 28 14:43:16 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606596, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:16.344691 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:16.331886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:33505 (local address 127.1.201.62:39161)
0928 14:43:16.332012 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:16.332017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:16.332951 (+ 934us) server_negotiation.cc:407] Connection header received
0928 14:43:16.334127 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:16.334132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:16.334193 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:16.334249 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:16.336371 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.336942 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.338316 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.338661 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.338771 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:16.341169 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:16.341172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:16.341174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:16.341196 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:16.342649 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:16.343148 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:16.343152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:16.343154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:16.343233 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:16.343572 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:16.343576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:16.343577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:16.343786 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:16.343893 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:16.344391 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 14:43:16.344512 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:43:16.345587 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:16.332323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39161 (local address 127.1.201.3:33505)
0928 14:43:16.332800 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:16.332851 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:16.333859 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:16.334881 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:16.334891 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:16.335260 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:16.336216 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:16.336239 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.337067 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.337069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:16.338191 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:16.338200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.338807 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.338811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:16.338981 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:16.339609 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:16.339627 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:16.341001 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:16.342781 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:16.342786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:16.342800 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:16.343045 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:16.343326 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:16.343330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:16.343332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:16.343473 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:16.343892 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:16.343899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:16.344155 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 14:43:16.344341 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 14:43:16.344678 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":143,"threads_started":1}
I20260928 14:43:16.346848 14573 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39161
I20260928 14:43:16.347081 14573 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:16.347505 14573 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
I20260928 14:43:16.348517 14089 ts_manager.cc:194] Registered new tserver with Master: bbee1c7f1eea420e8b1b235a1c9781e0 (127.1.201.3:34219)
I20260928 14:43:16.349152 14089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:33505
I20260928 14:43:16.350241 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:43:16.351014 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 14170
I20260928 14:43:16.357554 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 14309
I20260928 14:43:16.365971 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 14444
I20260928 14:43:16.372416 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 14057
I20260928 14:43:16.380678 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:39161
--webserver_interface=127.1.201.62
--webserver_port=40759
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:39161
--ranger_config_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:47503/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.201.20:47503/kms with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:16.490438 14576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:16.490697 14576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:16.490736 14576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:16.494673 14576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:43:16.494725 14576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:16.494745 14576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:16.494791 14576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:43:16.494818 14576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:43:16.500279 14576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.201.20:47503/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:39161
--ranger_config_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=40759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:16.501312 14576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:16.502377 14576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:16.510514 14582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:16.510541 14584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:16.510550 14576 server_base.cc:1034] running on GCE node
W20260928 14:43:16.510497 14581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:16.511354 14576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:16.512444 14576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:16.513635 14576 hybrid_clock.cc:648] HybridClock initialized: now 1790606596513621 us; error 28 us; skew 500 ppm
Sep 28 14:43:16 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606596, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:16.516448 14576 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:43:16.517264 14576 webserver.cc:460] Webserver started at http://127.1.201.62:40759/ using document root <none> and password file <none>
I20260928 14:43:16.517759 14576 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:16.517889 14576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:43:16 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606596, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:16.582176 14576 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 14:43:16.584504 14592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:16.585922 14576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:43:16.586061 14576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "2904a37fde2249db9fc69f70cb260894"
format_stamp: "Formatted at 2026-09-28 14:43:14 on dist-test-slave-0n27"
server_key: "d7be8c1472d848bca284ad1c939de2fe"
server_key_iv: "137220cc2ad4605eea4d80ed75fad61c"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:16.586618 14576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:16.613677 14576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:16.616815 14576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:16.617301 14576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:16.624539 14576 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:39161
I20260928 14:43:16.624542 14644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:39161 every 8 connection(s)
I20260928 14:43:16.625562 14576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:43:16.628261 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 14576
I20260928 14:43:16.628762 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:47503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:39851
--local_ip_for_outbound_sockets=127.1.201.1
--tserver_master_addrs=127.1.201.62:39161
--webserver_port=46053
--webserver_interface=127.1.201.1
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.201.20:47503/kms with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:43:16.630141 14645 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:43:16.631807 14645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: Bootstrap starting.
I20260928 14:43:16.647691 14645 log.cc:824] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:16.654103 14645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:43:16.654487 14645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: Bootstrap complete.
I20260928 14:43:16.658450 14645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } }
I20260928 14:43:16.658803 14645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2904a37fde2249db9fc69f70cb260894, State: Initialized, Role: FOLLOWER
I20260928 14:43:16.659267 14645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } }
I20260928 14:43:16.659394 14645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:43:16.659477 14645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:43:16.659575 14645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:43:16.661383 14645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } }
I20260928 14:43:16.661721 14645 leader_election.cc:302] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2904a37fde2249db9fc69f70cb260894; no voters:
I20260928 14:43:16.662062 14645 leader_election.cc:288] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:43:16.662194 14649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:43:16.662429 14649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [term 2 LEADER]: Becoming Leader. State: Replica: 2904a37fde2249db9fc69f70cb260894, State: Running, Role: LEADER
I20260928 14:43:16.662775 14649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } }
I20260928 14:43:16.663235 14645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:43:16.664131 14650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2904a37fde2249db9fc69f70cb260894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } } }
I20260928 14:43:16.664259 14650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:16.664587 14651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2904a37fde2249db9fc69f70cb260894. Latest consensus state: current_term: 2 leader_uuid: "2904a37fde2249db9fc69f70cb260894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2904a37fde2249db9fc69f70cb260894" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 39161 } } }
I20260928 14:43:16.664721 14651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894 [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:16.665033 14658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:43:16.665525 14645 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:43:16.670624 14658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:43:16.671618 14658 catalog_manager.cc:1259] Loaded cluster ID: 84e17fc89e8f4603a42cf771ca885ea7
I20260928 14:43:16.671721 14658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:43:16.674923 14658 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:43:16.675791 14658 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: Loaded TSK: 0
I20260928 14:43:16.677218 14658 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:43:16.768968 14647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:16.769373 14647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:16.769452 14647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:16.775413 14647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:16.775529 14647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:16.775671 14647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:43:16.783206 14647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.201.20:47503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:39851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=46053
--tserver_master_addrs=127.1.201.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:16.784513 14647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:16.785831 14647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:16.797591 14683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:16.797585 14684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:16.798014 14686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:16.803004 14647 server_base.cc:1034] running on GCE node
I20260928 14:43:16.803627 14647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:16.804441 14647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:16.805629 14647 hybrid_clock.cc:648] HybridClock initialized: now 1790606596805600 us; error 48 us; skew 500 ppm
Sep 28 14:43:16 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606596, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:16.809345 14647 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:43:16.810484 14647 webserver.cc:460] Webserver started at http://127.1.201.1:46053/ using document root <none> and password file <none>
I20260928 14:43:16.811189 14647 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:16.811288 14647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:43:16 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606596, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:16.841449 14647 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:43:16.844584 14694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:16.846019 14647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 14:43:16.846117 14647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "40f4090fbf0441a69ff455eefc90ced6"
format_stamp: "Formatted at 2026-09-28 14:43:15 on dist-test-slave-0n27"
server_key: "fb92333d310f4d80552707e4ba55ca72"
server_key_iv: "8024395e1e0cbc8703b70b9ce5fd9799"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:16.846750 14647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:16.879319 14647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:16.882824 14647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:16.883044 14647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:16.883674 14647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:16.884629 14647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:16.884750 14647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:16.884891 14647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:16.884979 14647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:16.896140 14647 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:39851
I20260928 14:43:16.896170 14808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:39851 every 8 connection(s)
I20260928 14:43:16.897359 14647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:43:16.907620 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 14647
I20260928 14:43:16.908108 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:47503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:38363
--local_ip_for_outbound_sockets=127.1.201.2
--tserver_master_addrs=127.1.201.62:39161
--webserver_port=42673
--webserver_interface=127.1.201.2
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.201.20:47503/kms with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:43:16 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606596, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:16.917838 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:16.899865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39161 (local address 127.1.201.1:34529)
0928 14:43:16.900968 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:16.901008 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:16.901940 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:16.903075 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:16.903083 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:16.903416 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:16.904069 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:16.904089 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.906002 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.906006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:16.907348 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:16.907359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.908427 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.908432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:16.908633 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:16.909353 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:16.909377 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:16.911209 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:16.913872 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:16.913880 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:16.913916 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:16.914298 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:16.914774 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:16.914780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:16.914783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:16.914965 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:16.915757 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:16.915767 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:16.916120 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 14:43:16.916348 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 14:43:16.916713 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":140,"threads_started":1}
I20260928 14:43:16.917858 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:16.899695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:34529 (local address 127.1.201.62:39161)
0928 14:43:16.900320 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:16.900342 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:16.901143 (+ 801us) server_negotiation.cc:407] Connection header received
0928 14:43:16.902378 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:16.902388 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:16.902684 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:16.903011 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:16.904265 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.905851 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.907497 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:16.908232 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:16.908414 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:16.911438 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:16.911443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:16.911455 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:16.911482 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:16.913459 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:16.914455 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:16.914464 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:16.914473 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:16.914625 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:16.915100 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:16.915104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:16.915106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:16.915589 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:16.915761 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:16.916527 (+ 766us) server_negotiation.cc:299] Negotiation successful
0928 14:43:16.916850 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":130,"threads_started":1}
I20260928 14:43:16.921404 14809 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39161
I20260928 14:43:16.921638 14809 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:16.922298 14809 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
W20260928 14:43:16.924561 14809 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:39161 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:43:16.924677 14809 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:16.924831 14809 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
I20260928 14:43:16.925704 14809 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:16.925907 14809 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
W20260928 14:43:16.926614 14809 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:43:17.035059 14813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:17.035414 14813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:17.035532 14813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:17.039712 14813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:17.039788 14813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:17.039882 14813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:43:17.045259 14813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.201.20:47503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:38363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=42673
--tserver_master_addrs=127.1.201.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:17.046378 14813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:17.047518 14813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:17.059821 14819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:17.060137 14821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:17.060384 14818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:17.060458 14813 server_base.cc:1034] running on GCE node
I20260928 14:43:17.061079 14813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:17.061821 14813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:17.065129 14813 hybrid_clock.cc:648] HybridClock initialized: now 1790606597065081 us; error 53 us; skew 500 ppm
Sep 28 14:43:17 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606597, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:17.069078 14813 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:43:17.070214 14813 webserver.cc:460] Webserver started at http://127.1.201.2:42673/ using document root <none> and password file <none>
I20260928 14:43:17.070958 14813 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:17.071115 14813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:43:17 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606597, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:17.097352 14813 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 14:43:17.100399 14830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:17.101850 14813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 14:43:17.101969 14813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "697d875a79c04d1fa263560555eb7eb4"
format_stamp: "Formatted at 2026-09-28 14:43:15 on dist-test-slave-0n27"
server_key: "5cd76df6502d39feaeb099ef781d2a79"
server_key_iv: "63a9496ff66d8c90788a5c9a2744df7d"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:17.102775 14813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:17.121742 14813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:17.124922 14813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:17.125129 14813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:17.125828 14813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:17.126729 14813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:17.126803 14813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:17.126865 14813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:17.126919 14813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:17.138144 14813 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:38363
I20260928 14:43:17.138176 14944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:38363 every 8 connection(s)
I20260928 14:43:17.139413 14813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:43:17.147925 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 14813
I20260928 14:43:17.148438 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.201.20:47503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:34219
--local_ip_for_outbound_sockets=127.1.201.3
--tserver_master_addrs=127.1.201.62:39161
--webserver_port=38629
--webserver_interface=127.1.201.3
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:35823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.201.20:47503/kms with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:43:17 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606597, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:17.157173 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:17.141472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:51045 (local address 127.1.201.62:39161)
0928 14:43:17.141612 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:17.141615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:17.142757 (+ 1142us) server_negotiation.cc:407] Connection header received
0928 14:43:17.144073 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:17.144079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:17.144141 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:17.144212 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:17.147371 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:17.147938 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:17.149444 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:17.149676 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:17.149829 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:17.152670 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:17.152675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:17.152680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:17.152711 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:17.154527 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:17.155206 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:17.155210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:17.155212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:17.155302 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:17.155751 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:17.155754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:17.155756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:17.155991 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:17.156121 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:17.156832 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 14:43:17.156980 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:43:17.158473 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:17.142014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39161 (local address 127.1.201.2:51045)
0928 14:43:17.142575 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:17.142633 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:17.143864 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:17.146148 (+ 2284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:17.146158 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:17.146541 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:17.147204 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:17.147225 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:17.148138 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:17.148141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:17.149312 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:17.149321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:17.149849 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:17.149853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:17.150030 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:17.150705 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:17.150730 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:17.152481 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:17.154658 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:17.154667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:17.154685 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:17.155067 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:17.155407 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:17.155412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:17.155417 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:17.155624 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:17.156135 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:17.156145 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:17.156503 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 14:43:17.156728 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 14:43:17.157103 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":153,"threads_started":1}
I20260928 14:43:17.159795 14945 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39161
I20260928 14:43:17.160071 14945 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:17.160703 14945 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
W20260928 14:43:17.162379 14945 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:39161 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:43:17.162497 14945 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:17.162645 14945 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
I20260928 14:43:17.163513 14945 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:17.163720 14945 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
W20260928 14:43:17.164580 14945 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:43:17.278612 14948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:17.278893 14948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:17.278998 14948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:17.282920 14948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:17.282990 14948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:17.283107 14948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:43:17.290192 14948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.201.20:47503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:34219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=38629
--tserver_master_addrs=127.1.201.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:17.291673 14948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:17.293040 14948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:17.302884 14956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:17.302914 14953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:17.302983 14954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:17.303794 14948 server_base.cc:1034] running on GCE node
I20260928 14:43:17.304191 14948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:17.304991 14948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:17.306283 14948 hybrid_clock.cc:648] HybridClock initialized: now 1790606597306237 us; error 43 us; skew 500 ppm
Sep 28 14:43:17 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606597, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:17.309588 14948 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:43:17.310595 14948 webserver.cc:460] Webserver started at http://127.1.201.3:38629/ using document root <none> and password file <none>
I20260928 14:43:17.311225 14948 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:17.311327 14948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:43:17 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606597, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:17.333633 14948 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260928 14:43:17.336745 14964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:17.338408 14948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:43:17.338526 14948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "bbee1c7f1eea420e8b1b235a1c9781e0"
format_stamp: "Formatted at 2026-09-28 14:43:16 on dist-test-slave-0n27"
server_key: "2f99d8333e3f6b5f02328442e745565b"
server_key_iv: "a943870873411fb2b35dbc5845ef347d"
server_key_version: "kuduclusterkey@0"
I20260928 14:43:17.339286 14948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:17.354588 14948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:17.357491 14948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:17.357686 14948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:17.358299 14948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:17.359223 14948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:17.359289 14948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:17.359377 14948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:17.359427 14948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:17.372001 14948 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:34219
I20260928 14:43:17.373339 14948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:43:17.380462 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 14948
I20260928 14:43:17.381042 15078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:34219 every 8 connection(s)
Sep 28 14:43:17 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606597, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:17.389894 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:17.377354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:37617 (local address 127.1.201.62:39161)
0928 14:43:17.377493 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:17.377498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:17.377515 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:43:17.378193 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:17.378198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:17.378258 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:17.378315 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:17.379926 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:17.380761 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:17.382898 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:17.383181 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:17.383243 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:17.385988 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:17.385991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:17.385994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:17.386017 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:17.387621 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:17.388180 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:17.388183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:17.388185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:17.388257 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:17.388612 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:17.388615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:17.388616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:17.388819 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:17.388905 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:17.389577 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 14:43:17.389698 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:43:17.390689 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:17.381142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.1.201.62:39161)
0928 14:43:17.381685 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:17.381689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:17.381700 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:43:17.381733 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:17.381735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:17.381802 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:17.382726 (+ 924us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:17.382810 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:17.383461 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:17.384676 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:17.384957 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:17.385070 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:17.386626 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:17.386630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:17.386640 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:17.386702 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:17.388096 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:17.388533 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:17.388538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:17.388543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:17.388621 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:17.388959 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:17.388962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:17.388964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:17.389150 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:17.389368 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:17.390395 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0928 14:43:17.390508 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":87,"threads_started":1}
I20260928 14:43:17.391021 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:17.376310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:39161 (local address 127.1.201.3:37617)
0928 14:43:17.376851 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:17.376906 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:17.377935 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:17.378732 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:17.378741 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:17.379075 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:17.379743 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:17.379764 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:17.381440 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:17.381445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:17.382797 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:17.382805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:17.383427 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:17.383430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:17.383618 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:17.384228 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:17.384243 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:17.385830 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:17.387717 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:17.387722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:17.387737 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:17.388060 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:17.388349 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:17.388352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:17.388353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:17.388509 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:17.388974 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:17.388980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:17.389288 (+ 308us) client_negotiation.cc:769] Sending connection context
0928 14:43:17.389526 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 14:43:17.389866 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":135,"threads_started":1}
I20260928 14:43:17.392980 15079 heartbeater.cc:347] Connected to a master server at 127.1.201.62:39161
I20260928 14:43:17.393236 15079 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:17.393986 15079 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
W20260928 14:43:17.395605 15079 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:39161 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:43:17.395726 15079 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:17.395896 15079 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
I20260928 14:43:17.396673 15079 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:17.396836 15079 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
W20260928 14:43:17.397552 15079 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 14:43:17 dist-test-slave-0n27 krb5kdc[13613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606597, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:17.926818 14809 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:17.927134 14809 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
I20260928 14:43:17.928148 14603 ts_manager.cc:194] Registered new tserver with Master: 40f4090fbf0441a69ff455eefc90ced6 (127.1.201.1:39851)
I20260928 14:43:17.929346 14603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:34529
Sep 28 14:43:18 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606597, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:18.164779 14945 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:18.165067 14945 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
I20260928 14:43:18.165817 14603 ts_manager.cc:194] Registered new tserver with Master: 697d875a79c04d1fa263560555eb7eb4 (127.1.201.2:38363)
I20260928 14:43:18.166623 14603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:51045
Sep 28 14:43:18 dist-test-slave-0n27 krb5kdc[13613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606597, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for HTTP/127.1.201.20@KRBTEST.COM
I20260928 14:43:18.397787 15079 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:18.398099 15079 heartbeater.cc:511] Master 127.1.201.62:39161 requested a full tablet report, sending...
I20260928 14:43:18.398897 14603 ts_manager.cc:194] Registered new tserver with Master: bbee1c7f1eea420e8b1b235a1c9781e0 (127.1.201.3:34219)
I20260928 14:43:18.399679 14603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:37617
I20260928 14:43:18.403543 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:43:18.423429 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.405603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.1.201.62:39161)
0928 14:43:18.406250 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:18.406255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:18.412870 (+ 6615us) server_negotiation.cc:407] Connection header received
0928 14:43:18.412926 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:18.412931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:18.412996 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:18.413082 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:18.414102 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.414978 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.416216 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.416601 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.416719 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.418801 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:18.418806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:18.418809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:18.418836 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:18.420966 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:18.421415 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:18.421419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:18.421420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:18.421505 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:18.421861 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:18.421864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:18.421866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:18.422092 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:18.422235 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:18.422982 (+ 747us) server_negotiation.cc:299] Negotiation successful
0928 14:43:18.423116 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":102,"threads_started":1}
I20260928 14:43:18.427847 14603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:43:18.433601 14603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:43:18.456936 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.447771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:39851 (local address 127.0.0.1:48670)
0928 14:43:18.450795 (+ 3024us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:18.450848 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:18.450996 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:18.451827 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:18.451833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:18.451859 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:18.452006 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:18.452013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.453829 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.453833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:18.455279 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:18.455290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.456406 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.456410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:18.456520 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.456564 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:43:18.456646 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:43:18.456774 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2836,"mutex_wait_us":32,"thread_start_us":107,"threads_started":1}
I20260928 14:43:18.457995 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.448453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.1.201.1:39851)
0928 14:43:18.448758 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:18.448766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:18.451306 (+ 2540us) server_negotiation.cc:407] Connection header received
0928 14:43:18.451400 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:18.451406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:18.451628 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:18.452394 (+ 766us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:18.452550 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.453684 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.455419 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.456271 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.457532 (+ 1261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.457696 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:18.457747 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:43:18.457862 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260928 14:43:18.458182 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.450250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:38363 (local address 127.0.0.1:51108)
0928 14:43:18.451482 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:18.451493 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:18.451552 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:18.452302 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:18.452305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:18.452323 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:18.452418 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:18.452424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.454780 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.454783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:18.455997 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:18.456008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.457769 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.457773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:18.457884 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.457910 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:43:18.457964 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 14:43:18.458032 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260928 14:43:18.459048 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.450428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.1.201.2:38363)
0928 14:43:18.451848 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:18.451854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:18.451876 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:43:18.451949 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:18.451954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:18.452137 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:18.453336 (+ 1199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:18.453521 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.454649 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.456863 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.457628 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.458610 (+ 982us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.458752 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:18.458805 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 14:43:18.458898 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":120,"threads_started":1}
I20260928 14:43:18.461239 14727 tablet_service.cc:1467] Processing CreateTablet for tablet 1301f2b612c24824a531ca869fbaaf90 (DEFAULT_TABLE table=test-table [id=06b6780cb98f4ef999b3126982b3acac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:18.462388 14727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1301f2b612c24824a531ca869fbaaf90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:18.468645 14877 tablet_service.cc:1467] Processing CreateTablet for tablet 1301f2b612c24824a531ca869fbaaf90 (DEFAULT_TABLE table=test-table [id=06b6780cb98f4ef999b3126982b3acac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:18.469751 14877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1301f2b612c24824a531ca869fbaaf90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:18.478045 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.449168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:34219 (local address 127.0.0.1:60886)
0928 14:43:18.451243 (+ 2075us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:18.451255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:18.451330 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:18.460573 (+ 9243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:18.460578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:18.460599 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:18.460716 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:18.460722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.475344 (+ 14622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.475348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:18.476566 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:18.476578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.477605 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.477611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:18.477710 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.477740 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:43:18.477833 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:43:18.477909 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2002,"thread_start_us":117,"threads_started":1}
I20260928 14:43:18.478755 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.449641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.1.201.3:34219)
0928 14:43:18.460096 (+ 10455us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:18.460105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:18.460131 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:43:18.460214 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:18.460219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:18.460422 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:18.473936 (+ 13514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:18.474139 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.475202 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.476740 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.477484 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.478311 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.478469 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:18.478519 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 14:43:18.478607 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10338,"thread_start_us":90,"threads_started":1}
I20260928 14:43:18.481729 15013 tablet_service.cc:1467] Processing CreateTablet for tablet 1301f2b612c24824a531ca869fbaaf90 (DEFAULT_TABLE table=test-table [id=06b6780cb98f4ef999b3126982b3acac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:18.482851 15013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1301f2b612c24824a531ca869fbaaf90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:18.498637 15114 tablet_bootstrap.cc:492] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4: Bootstrap starting.
I20260928 14:43:18.501137 15114 tablet_bootstrap.cc:654] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:18.501543 15115 tablet_bootstrap.cc:492] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: Bootstrap starting.
I20260928 14:43:18.501964 15116 tablet_bootstrap.cc:492] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6: Bootstrap starting.
I20260928 14:43:18.505136 15114 log.cc:824] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:18.506229 15116 tablet_bootstrap.cc:654] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:18.507229 15115 tablet_bootstrap.cc:654] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:18.509708 15116 log.cc:824] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:18.510435 15115 log.cc:824] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:18.514906 15114 tablet_bootstrap.cc:492] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4: No bootstrap required, opened a new log
I20260928 14:43:18.515244 15114 ts_tablet_manager.cc:1397] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260928 14:43:18.519585 15114 raft_consensus.cc:348] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 } } peers { permanent_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34219 } } peers { permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 } }
I20260928 14:43:18.521867 15115 tablet_bootstrap.cc:492] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: No bootstrap required, opened a new log
I20260928 14:43:18.522121 15115 ts_tablet_manager.cc:1397] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260928 14:43:18.526288 15115 raft_consensus.cc:348] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 } } peers { permanent_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34219 } } peers { permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 } }
I20260928 14:43:18.526639 15115 raft_consensus.cc:374] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:18.526737 15115 raft_consensus.cc:729] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbee1c7f1eea420e8b1b235a1c9781e0, State: Initialized, Role: FOLLOWER
I20260928 14:43:18.527086 15114 raft_consensus.cc:374] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:18.527295 15114 raft_consensus.cc:729] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697d875a79c04d1fa263560555eb7eb4, State: Initialized, Role: FOLLOWER
I20260928 14:43:18.527673 15115 consensus_queue.cc:260] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 } } peers { permanent_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34219 } } peers { permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 } }
I20260928 14:43:18.527774 15116 tablet_bootstrap.cc:492] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6: No bootstrap required, opened a new log
I20260928 14:43:18.527978 15116 ts_tablet_manager.cc:1397] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20260928 14:43:18.528414 15114 consensus_queue.cc:260] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 } } peers { permanent_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34219 } } peers { permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 } }
I20260928 14:43:18.529599 15115 ts_tablet_manager.cc:1428] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:43:18.530619 15114 ts_tablet_manager.cc:1428] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260928 14:43:18.531163 15079 heartbeater.cc:503] Master 127.1.201.62:39161 was elected leader, sending a full tablet report...
I20260928 14:43:18.532174 15116 raft_consensus.cc:348] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 } } peers { permanent_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34219 } } peers { permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 } }
I20260928 14:43:18.532433 15116 raft_consensus.cc:374] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:18.532478 15116 raft_consensus.cc:729] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f4090fbf0441a69ff455eefc90ced6, State: Initialized, Role: FOLLOWER
I20260928 14:43:18.532881 15116 consensus_queue.cc:260] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 } } peers { permanent_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34219 } } peers { permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 } }
I20260928 14:43:18.533748 15116 ts_tablet_manager.cc:1428] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:43:18.534082 14809 heartbeater.cc:503] Master 127.1.201.62:39161 was elected leader, sending a full tablet report...
I20260928 14:43:18.538978 14945 heartbeater.cc:503] Master 127.1.201.62:39161 was elected leader, sending a full tablet report...
I20260928 14:43:18.581818 15120 raft_consensus.cc:482] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:43:18.582067 15120 raft_consensus.cc:504] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 } } peers { permanent_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34219 } } peers { permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 } }
I20260928 14:43:18.583479 15120 leader_election.cc:288] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40f4090fbf0441a69ff455eefc90ced6 (127.1.201.1:39851), 697d875a79c04d1fa263560555eb7eb4 (127.1.201.2:38363)
I20260928 14:43:18.594594 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.584466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:53251 (local address 127.1.201.1:39851)
0928 14:43:18.584593 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:18.584598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:18.590075 (+ 5477us) server_negotiation.cc:407] Connection header received
0928 14:43:18.590322 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:18.590326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:18.590386 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:18.590439 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:18.590934 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.591636 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.593251 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.593878 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.593957 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.594028 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:18.594376 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 14:43:18.594459 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:43:18.595249 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.583897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:39851 (local address 127.1.201.3:53251)
0928 14:43:18.589964 (+ 6067us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:18.589981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:18.590172 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:18.590599 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:18.590602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:18.590622 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:18.590802 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:18.590809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.591801 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.591805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:18.593103 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:18.593115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.594122 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.594125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:18.594234 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.594257 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:43:18.595022 (+ 765us) client_negotiation.cc:240] Negotiation successful
0928 14:43:18.595104 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5981,"thread_start_us":153,"threads_started":1}
I20260928 14:43:18.596644 14763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1301f2b612c24824a531ca869fbaaf90" candidate_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40f4090fbf0441a69ff455eefc90ced6" is_pre_election: true
I20260928 14:43:18.597002 14763 raft_consensus.cc:2454] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbee1c7f1eea420e8b1b235a1c9781e0 in term 0.
I20260928 14:43:18.597713 14965 leader_election.cc:302] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40f4090fbf0441a69ff455eefc90ced6, bbee1c7f1eea420e8b1b235a1c9781e0; no voters:
I20260928 14:43:18.598058 15120 raft_consensus.cc:2781] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:43:18.598160 15120 raft_consensus.cc:482] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:43:18.598225 15120 raft_consensus.cc:3037] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:18.599597 15120 raft_consensus.cc:504] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 } } peers { permanent_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34219 } } peers { permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 } }
I20260928 14:43:18.599967 15120 leader_election.cc:288] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [CANDIDATE]: Term 1 election: Requested vote from peers 40f4090fbf0441a69ff455eefc90ced6 (127.1.201.1:39851), 697d875a79c04d1fa263560555eb7eb4 (127.1.201.2:38363)
I20260928 14:43:18.600528 14763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1301f2b612c24824a531ca869fbaaf90" candidate_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40f4090fbf0441a69ff455eefc90ced6"
I20260928 14:43:18.600651 14763 raft_consensus.cc:3037] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:18.601936 14763 raft_consensus.cc:2454] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbee1c7f1eea420e8b1b235a1c9781e0 in term 1.
I20260928 14:43:18.602380 14965 leader_election.cc:302] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40f4090fbf0441a69ff455eefc90ced6, bbee1c7f1eea420e8b1b235a1c9781e0; no voters:
I20260928 14:43:18.602591 15120 raft_consensus.cc:2781] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:43:18.602810 15120 raft_consensus.cc:686] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 1 LEADER]: Becoming Leader. State: Replica: bbee1c7f1eea420e8b1b235a1c9781e0, State: Running, Role: LEADER
I20260928 14:43:18.603174 15120 consensus_queue.cc:237] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 } } peers { permanent_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34219 } } peers { permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 } }
I20260928 14:43:18.607193 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.584966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:38363 (local address 127.1.201.3:50319)
0928 14:43:18.595868 (+ 10902us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:18.595884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:18.597936 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:18.598371 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:18.598376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:18.598410 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:18.598547 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:18.598555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.599578 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.599582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:18.600667 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:18.600677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.606734 (+ 6057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.606737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:18.606841 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.606865 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:43:18.606947 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:43:18.607044 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10829,"thread_start_us":98,"threads_started":1}
I20260928 14:43:18.607211 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.585389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:50319 (local address 127.1.201.2:38363)
0928 14:43:18.585508 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:18.585512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:18.595967 (+ 10455us) server_negotiation.cc:407] Connection header received
0928 14:43:18.598096 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:18.598100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:18.598164 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:18.598222 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:18.598681 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.599413 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.605891 (+ 6478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.606467 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.606555 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.606622 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:18.606988 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 14:43:18.607083 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:43:18.607124 14603 catalog_manager.cc:5495] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbee1c7f1eea420e8b1b235a1c9781e0 (127.1.201.3). New cstate: current_term: 1 leader_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260928 14:43:18.608762 14880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1301f2b612c24824a531ca869fbaaf90" candidate_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697d875a79c04d1fa263560555eb7eb4" is_pre_election: true
I20260928 14:43:18.609028 14889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1301f2b612c24824a531ca869fbaaf90" candidate_uuid: "bbee1c7f1eea420e8b1b235a1c9781e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697d875a79c04d1fa263560555eb7eb4"
I20260928 14:43:18.609074 14880 raft_consensus.cc:2454] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbee1c7f1eea420e8b1b235a1c9781e0 in term 0.
W20260928 14:43:18.625233 15080 tablet.cc:2369] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:43:18.641999 14946 tablet.cc:2369] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:43:18.645424 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.639226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.1.201.3:34219)
0928 14:43:18.639352 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:18.639357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:18.639370 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:43:18.639546 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:18.639549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:18.639626 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:18.639747 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:43:18.640077 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.640896 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.643137 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:18.643490 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:18.643599 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:18.644062 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:43:18.644637 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:43:18.644757 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:18.645196 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 14:43:18.645257 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:43:18.652181 14889 raft_consensus.cc:3037] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:18.652308 14763 raft_consensus.cc:1261] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6 [term 1 FOLLOWER]: Refusing update from remote peer bbee1c7f1eea420e8b1b235a1c9781e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 14:43:18.652570 14810 tablet.cc:2369] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:43:18.653141 15120 consensus_queue.cc:1035] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40f4090fbf0441a69ff455eefc90ced6" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 39851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:43:18.653836 14889 raft_consensus.cc:1261] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4 [term 1 FOLLOWER]: Refusing update from remote peer bbee1c7f1eea420e8b1b235a1c9781e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:43:18.654587 15120 consensus_queue.cc:1035] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "697d875a79c04d1fa263560555eb7eb4" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 38363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:43:18.663897 15128 mvcc.cc:205] Tried to move back new op lower bound from 7334324628071849984 to 7334324627882790912. Current Snapshot: MvccSnapshot[applied={T|T < 7334324628071849984}]
I20260928 14:43:18.670415 15133 mvcc.cc:205] Tried to move back new op lower bound from 7334324628071849984 to 7334324627882790912. Current Snapshot: MvccSnapshot[applied={T|T < 7334324628071849984}]
I20260928 14:43:18.676159 14603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:43:18.676337 14603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:43:18.679042 14603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2904a37fde2249db9fc69f70cb260894: Sending DeleteTablet for 3 replicas of tablet 1301f2b612c24824a531ca869fbaaf90
I20260928 14:43:18.679826 14727 tablet_service.cc:1514] Processing DeleteTablet for tablet 1301f2b612c24824a531ca869fbaaf90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:18 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:48670
I20260928 14:43:18.680325 15145 tablet_replica.cc:330] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6: stopping tablet replica
I20260928 14:43:18.680619 14877 tablet_service.cc:1514] Processing DeleteTablet for tablet 1301f2b612c24824a531ca869fbaaf90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:18 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:51108
I20260928 14:43:18.680712 15145 raft_consensus.cc:2229] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:43:18.681010 15145 raft_consensus.cc:2258] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:43:18.681535 15013 tablet_service.cc:1514] Processing DeleteTablet for tablet 1301f2b612c24824a531ca869fbaaf90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:18 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:60886
I20260928 14:43:18.682611 15145 ts_tablet_manager.cc:1905] T 1301f2b612c24824a531ca869fbaaf90 P 40f4090fbf0441a69ff455eefc90ced6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:43:18.682961 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 14647
I20260928 14:43:18.690378 15147 tablet_replica.cc:330] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: stopping tablet replica
I20260928 14:43:18.690824 15147 raft_consensus.cc:2229] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:43:18.691262 15147 raft_consensus.cc:2258] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:43:18.691542 15146 tablet_replica.cc:330] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4: stopping tablet replica
W20260928 14:43:18.691768 14593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:39851 (error 108)
I20260928 14:43:18.692196 15146 raft_consensus.cc:2229] T 1301f2b612c24824a531ca869fbaaf90 P 697d875a79c04d1fa263560555eb7eb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:43:18.692656 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 14813
I20260928 14:43:18.694545 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.694278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:39851 (local address 127.0.0.1:48676)
0928 14:43:18.694389 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:18.694478 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:43:18.694754 14593 catalog_manager.cc:4584] TS 40f4090fbf0441a69ff455eefc90ced6 (127.1.201.1:39851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1301f2b612c24824a531ca869fbaaf90: Network error: Client connection negotiation failed: client connection to 127.1.201.1:39851: connect: Connection refused (error 111)
I20260928 14:43:18.697939 15147 ts_tablet_manager.cc:1905] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:43:18.700747 15147 ts_tablet_manager.cc:1918] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:43:18.700899 15147 log.cc:1196] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/1301f2b612c24824a531ca869fbaaf90
I20260928 14:43:18.701299 15147 ts_tablet_manager.cc:1939] T 1301f2b612c24824a531ca869fbaaf90 P bbee1c7f1eea420e8b1b235a1c9781e0: Deleting consensus metadata
I20260928 14:43:18.702759 14595 catalog_manager.cc:4841] TS bbee1c7f1eea420e8b1b235a1c9781e0 (127.1.201.3:34219): tablet 1301f2b612c24824a531ca869fbaaf90 (table test-table [id=06b6780cb98f4ef999b3126982b3acac]) successfully deleted
I20260928 14:43:18.703106 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 14948
I20260928 14:43:18.703859 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:18.703468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:38363 (local address 127.0.0.1:51120)
0928 14:43:18.703692 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:18.703756 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.2:38363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 14:43:18.709100 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 14576
I20260928 14:43:18.717576 1828 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:43:19.052959 1828 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:43:19.053092 1828 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:43:19.397887 1828 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:43:19.398 UTC [13643] LOG: received smart shutdown request
2026-09-28 14:43:19.401 UTC [13643] LOG: background worker "logical replication launcher" (PID 13652) exited with exit code 1
2026-09-28 14:43:19.402 UTC [13647] LOG: shutting down
2026-09-28 14:43:19.555 UTC [13643] LOG: database system is shut down
2026-09-28T14:43:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47582 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:43:19 dist-test-slave-0n27 krb5kdc[15157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:43:19 dist-test-slave-0n27 krb5kdc[15157](info): set up 2 sockets
Sep 28 14:43:19 dist-test-slave-0n27 krb5kdc[15157](info): commencing operation
krb5kdc: starting...
W20260928 14:43:20.710758 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:43:20 dist-test-slave-0n27 krb5kdc[15157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:43:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:43:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:43:20.829093 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:44763
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:44763,127.1.201.61:46159,127.1.201.60:37727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:20.938499 15170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:20.938805 15170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:20.938881 15170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:20.942814 15170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:43:20.942902 15170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:20.942937 15170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:20.942996 15170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:43:20.943048 15170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:43:20.948365 15170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:44763,127.1.201.61:46159,127.1.201.60:37727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:44763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:20.949424 15170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:20.950522 15170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:20.959056 15175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:20.959057 15176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:20.959057 15178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:20.959113 15170 server_base.cc:1034] running on GCE node
I20260928 14:43:20.959869 15170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:20.960975 15170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:20.962157 15170 hybrid_clock.cc:648] HybridClock initialized: now 1790606600962143 us; error 37 us; skew 500 ppm
Sep 28 14:43:20 dist-test-slave-0n27 krb5kdc[15157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606600, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:20.964777 15170 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:43:20.965653 15170 webserver.cc:460] Webserver started at http://127.1.201.62:39237/ using document root <none> and password file <none>
I20260928 14:43:20.966259 15170 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:20.966349 15170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:20.966605 15170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:20.968622 15170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "2c37907173234392852ef9554f7bfc87"
format_stamp: "Formatted at 2026-09-28 14:43:20 on dist-test-slave-0n27"
server_key: "d413057c772127d6883ddb8fb33ab5a2"
server_key_iv: "c9b2061e77e08f1acbf217fcebdb58d9"
server_key_version: "encryptionkey@0"
I20260928 14:43:20.969200 15170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "2c37907173234392852ef9554f7bfc87"
format_stamp: "Formatted at 2026-09-28 14:43:20 on dist-test-slave-0n27"
server_key: "d413057c772127d6883ddb8fb33ab5a2"
server_key_iv: "c9b2061e77e08f1acbf217fcebdb58d9"
server_key_version: "encryptionkey@0"
I20260928 14:43:20.972795 15170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 14:43:20.975250 15185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:20.976361 15170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 14:43:20.976495 15170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "2c37907173234392852ef9554f7bfc87"
format_stamp: "Formatted at 2026-09-28 14:43:20 on dist-test-slave-0n27"
server_key: "d413057c772127d6883ddb8fb33ab5a2"
server_key_iv: "c9b2061e77e08f1acbf217fcebdb58d9"
server_key_version: "encryptionkey@0"
I20260928 14:43:20.976603 15170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:21.011278 15170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:21.014132 15170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:21.014348 15170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:21.021384 15170 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:44763
I20260928 14:43:21.021409 15237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:44763 every 8 connection(s)
I20260928 14:43:21.022341 15170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:43:21.025038 15238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:21.025381 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 15170
I20260928 14:43:21.025501 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:43:21.025833 1828 external_mini_cluster.cc:1428] Setting key fe392f565d0b0dfca217f1a599109f88
I20260928 14:43:21.028826 15238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.201.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.61@KRBTEST.COM" created.
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606600, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:21.044734 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.031138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.1.201.62:44763)
0928 14:43:21.031662 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.031679 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.031852 (+ 173us) server_negotiation.cc:407] Connection header received
0928 14:43:21.033185 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.033189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.033357 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.033527 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.034365 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.035291 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.036614 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.037274 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.037463 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.040041 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.040048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.040066 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.040085 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.041627 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.042117 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.042123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.042125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.042229 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.042555 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.042558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.042560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.042932 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.043084 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.043439 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.043687 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":409,"spinlock_wait_cycles":49408,"thread_start_us":146,"threads_started":1}
I20260928 14:43:21.044734 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.031138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44763 (local address 127.0.0.1:60516)
0928 14:43:21.031680 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.031736 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:21.032874 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:21.033566 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:21.033572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:21.033986 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:21.034238 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.034251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.035426 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.035429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.036500 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.036509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.037434 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.037437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.037563 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.038235 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:21.038259 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:21.039900 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:21.041758 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.041767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.041796 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.042013 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.042346 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.042349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.042351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.042460 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.043058 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:21.043064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:21.043211 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 14:43:21.043369 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 14:43:21.043730 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":142,"thread_start_us":138,"threads_started":1}
I20260928 14:43:21.047127 15238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:43:21.047824 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.047567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46159 (local address 127.0.0.1:41012)
0928 14:43:21.047682 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.047764 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:43:21.048009 15189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.201.61:46159: connect: Connection refused (error 111)
I20260928 14:43:21.048792 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.048582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46159 (local address 127.0.0.1:41020)
0928 14:43:21.048689 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.048733 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:43:21.048920 15238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.61:46159: Network error: Client connection negotiation failed: client connection to 127.1.201.61:46159: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab.
Entry for principal kudu/127.1.201.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab.
I20260928 14:43:21.050921 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.61:46159
--webserver_interface=127.1.201.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab
--principal=kudu/127.1.201.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:44763,127.1.201.61:46159,127.1.201.60:37727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:43:21.098172 15238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } attempt: 1
I20260928 14:43:21.099128 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.098838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46159 (local address 127.0.0.1:41034)
0928 14:43:21.098987 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.099055 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 14:43:21.099946 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.099728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46159 (local address 127.0.0.1:41036)
0928 14:43:21.099837 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.099884 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:43:21.100121 15238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.61:46159: Network error: Client connection negotiation failed: client connection to 127.1.201.61:46159: connect: Connection refused (error 111)
W20260928 14:43:21.164516 15245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:21.164777 15245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:21.164853 15245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:43:21.168299 15238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } attempt: 2
W20260928 14:43:21.168959 15245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:43:21.169005 15245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:21.169026 15245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:21.169062 15245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:43:21.169092 15245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:43:21.169123 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.168794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46159 (local address 127.0.0.1:41052)
0928 14:43:21.168939 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.169023 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 14:43:21.169909 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.169638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46159 (local address 127.0.0.1:41058)
0928 14:43:21.169774 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.169845 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 14:43:21.170089 15238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.61:46159: Network error: Client connection negotiation failed: client connection to 127.1.201.61:46159: connect: Connection refused (error 111)
I20260928 14:43:21.174583 15245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:44763,127.1.201.61:46159,127.1.201.60:37727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab
--principal=kudu/127.1.201.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.61:46159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.201.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:21.175658 15245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:21.176723 15245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:21.185348 15253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:21.185348 15250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:21.185351 15251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:21.185672 15245 server_base.cc:1034] running on GCE node
I20260928 14:43:21.186273 15245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:21.187326 15245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:21.188481 15245 hybrid_clock.cc:648] HybridClock initialized: now 1790606601188468 us; error 28 us; skew 500 ppm
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606601, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:21.191244 15245 init.cc:377] Logged in from keytab as kudu/127.1.201.61@KRBTEST.COM (short username kudu)
I20260928 14:43:21.192024 15245 webserver.cc:460] Webserver started at http://127.1.201.61:33609/ using document root <none> and password file <none>
I20260928 14:43:21.192528 15245 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:21.192585 15245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:21.192762 15245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:21.194761 15245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/data/instance:
uuid: "a74402dfc46a45e7910bf6e205c4a79d"
format_stamp: "Formatted at 2026-09-28 14:43:21 on dist-test-slave-0n27"
server_key: "69889a4a6d0a6e1705b937aa52464860"
server_key_iv: "96aa6c8a97a12b8ef2a90a7f3bb6be7d"
server_key_version: "encryptionkey@0"
I20260928 14:43:21.195267 15245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/wal/instance:
uuid: "a74402dfc46a45e7910bf6e205c4a79d"
format_stamp: "Formatted at 2026-09-28 14:43:21 on dist-test-slave-0n27"
server_key: "69889a4a6d0a6e1705b937aa52464860"
server_key_iv: "96aa6c8a97a12b8ef2a90a7f3bb6be7d"
server_key_version: "encryptionkey@0"
I20260928 14:43:21.198892 15245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 14:43:21.201229 15260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:21.202510 15245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:43:21.202644 15245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/wal
uuid: "a74402dfc46a45e7910bf6e205c4a79d"
format_stamp: "Formatted at 2026-09-28 14:43:21 on dist-test-slave-0n27"
server_key: "69889a4a6d0a6e1705b937aa52464860"
server_key_iv: "96aa6c8a97a12b8ef2a90a7f3bb6be7d"
server_key_version: "encryptionkey@0"
I20260928 14:43:21.202761 15245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:21.216007 15245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:21.218729 15245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:21.218916 15245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:21.225415 15245 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.61:46159
I20260928 14:43:21.225483 15313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.61:46159 every 8 connection(s)
I20260928 14:43:21.226392 15245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
I20260928 14:43:21.227317 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 15245
I20260928 14:43:21.227468 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-1/wal/instance
I20260928 14:43:21.227746 1828 external_mini_cluster.cc:1428] Setting key 43a2b0604720443d2f931d80786c624a
I20260928 14:43:21.229564 15314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:21.233398 15314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.201.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.60@KRBTEST.COM" created.
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606601, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:21.247794 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.235255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.1.201.62:44763)
0928 14:43:21.235422 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.235427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.236210 (+ 783us) server_negotiation.cc:407] Connection header received
0928 14:43:21.237678 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.237681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.237733 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.237869 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.239421 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.239927 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.241253 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.241534 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.241617 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.244102 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.244109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.244120 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.244169 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.245591 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.246209 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.246212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.246213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.246283 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.246668 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.246670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.246671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.246832 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.246904 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.247556 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.247657 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:43:21.249008 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.235636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44763 (local address 127.0.0.1:60524)
0928 14:43:21.236069 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.236109 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:21.237421 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:21.238063 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:21.238075 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:21.238562 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:21.239273 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.239292 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.240052 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.240055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.241103 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.241112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.241671 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.241673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.241857 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.242398 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:21.242414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:21.243929 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:21.245702 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.245711 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.245728 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.246108 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.246375 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.246381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.246386 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.246574 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.246939 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:21.246948 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:21.247281 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 14:43:21.247507 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 14:43:21.247866 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260928 14:43:21.250486 15314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606601, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab.
Entry for principal kudu/127.1.201.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab.
I20260928 14:43:21.259235 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.60:37727
--webserver_interface=127.1.201.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab
--principal=kudu/127.1.201.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:44763,127.1.201.61:46159,127.1.201.60:37727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:43:21.259684 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.250997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46159 (local address 127.0.0.1:41072)
0928 14:43:21.251451 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.251466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:21.251558 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:21.251916 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:21.251919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:21.252097 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:21.252183 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.252188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.253173 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.253176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.253851 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.253861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.254467 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.254470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.254571 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.255028 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:21.255039 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:21.256290 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:21.258064 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.258066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.258068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.258363 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.258669 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.258671 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.258672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.258745 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.259212 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:21.259215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:21.259300 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 14:43:21.259408 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:43:21.259555 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408}
I20260928 14:43:21.260167 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.251115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.1.201.61:46159)
0928 14:43:21.251452 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.251465 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.251529 (+ 64us) server_negotiation.cc:407] Connection header received
0928 14:43:21.251661 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.251670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.251810 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.251949 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.252295 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.253063 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.253964 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.254355 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.254471 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.256409 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.256417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.256426 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.256447 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.257972 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.258467 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.258472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.258473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.258562 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.258859 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.258864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.258866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.259102 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.259238 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.259927 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.260035 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":161,"threads_started":1}
I20260928 14:43:21.261256 15238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } attempt: 3
I20260928 14:43:21.262081 15314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:43:21.262856 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.262570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50800)
0928 14:43:21.262693 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.262780 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 14:43:21.263046 15263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
I20260928 14:43:21.264482 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.264216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50814)
0928 14:43:21.264342 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.264404 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 14:43:21.264757 15314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:37727: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606600, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
I20260928 14:43:21.270469 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.261899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.1.201.61:46159)
0928 14:43:21.262041 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.262047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.262063 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:43:21.262101 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.262119 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.262165 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.262284 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.263216 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.264025 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.264949 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.265170 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.265275 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.267454 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.267458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.267462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.267487 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.268843 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.269267 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.269271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.269273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.269347 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.269614 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.269619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.269621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.269857 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.269963 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.270237 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.270336 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:43:21.270471 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.261723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46159 (local address 127.0.0.1:41076)
0928 14:43:21.261886 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.261901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:21.261982 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:21.262310 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:21.262313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:21.262497 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:21.263081 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.263092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.264165 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.264168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.264804 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.264814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.265288 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.265291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.265397 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.265904 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:21.265920 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:21.267328 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:21.268967 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.268971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.268974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.269171 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.269438 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.269442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.269444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.269527 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.269964 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:21.269967 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:21.270065 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 14:43:21.270185 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 14:43:21.270316 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 14:43:21.271292 15238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:43:21.271936 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.271711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50824)
0928 14:43:21.271822 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.271871 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:43:21.272608 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.272391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50840)
0928 14:43:21.272528 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.272567 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 14:43:21.272907 15238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:37727: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
I20260928 14:43:21.304078 15238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } attempt: 1
I20260928 14:43:21.304946 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.304649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50844)
0928 14:43:21.304809 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.304870 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 14:43:21.305651 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.305470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50858)
0928 14:43:21.305564 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.305604 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 14:43:21.305833 15238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:37727: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
I20260928 14:43:21.314004 15314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } attempt: 1
I20260928 14:43:21.314755 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.314490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50872)
0928 14:43:21.314623 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.314690 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 14:43:21.315440 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.315261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50886)
0928 14:43:21.315351 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.315392 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:43:21.315642 15314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:37727: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
W20260928 14:43:21.373554 15320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:21.373857 15320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:21.373960 15320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:21.377993 15320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:43:21.378067 15320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:21.378103 15320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:21.378155 15320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:43:21.378213 15320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:43:21.381021 15238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } attempt: 2
I20260928 14:43:21.381875 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.381533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50892)
0928 14:43:21.381684 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.381745 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 14:43:21.382594 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.382354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50898)
0928 14:43:21.382461 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.382520 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:43:21.382745 15238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:37727: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
I20260928 14:43:21.383806 15314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } attempt: 2
I20260928 14:43:21.384187 15320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:44763,127.1.201.61:46159,127.1.201.60:37727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab
--principal=kudu/127.1.201.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.60:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.201.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:21.384521 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.384220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50908)
0928 14:43:21.384382 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.384452 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:43:21.385344 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.385113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50922)
0928 14:43:21.385234 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.385291 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:43:21.385483 15320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260928 14:43:21.385531 15314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:37727: Network error: Client connection negotiation failed: client connection to 127.1.201.60:37727: connect: Connection refused (error 111)
I20260928 14:43:21.386585 15320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:21.394842 15327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:21.394860 15326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:21.394843 15329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:21.395167 15320 server_base.cc:1034] running on GCE node
I20260928 14:43:21.395751 15320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:21.396848 15320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:21.398032 15320 hybrid_clock.cc:648] HybridClock initialized: now 1790606601398020 us; error 33 us; skew 500 ppm
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606601, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:21.400743 15320 init.cc:377] Logged in from keytab as kudu/127.1.201.60@KRBTEST.COM (short username kudu)
I20260928 14:43:21.401569 15320 webserver.cc:460] Webserver started at http://127.1.201.60:33605/ using document root <none> and password file <none>
I20260928 14:43:21.402163 15320 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:21.402253 15320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:21.402482 15320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:21.404543 15320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/data/instance:
uuid: "cdd9517c44284386a8887b667b58c5a4"
format_stamp: "Formatted at 2026-09-28 14:43:21 on dist-test-slave-0n27"
server_key: "41e502bf6c2a186401a9b8a0ac6b97af"
server_key_iv: "8755b49ef2ed0fbc0c255aedc7546e7d"
server_key_version: "encryptionkey@0"
I20260928 14:43:21.405119 15320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/wal/instance:
uuid: "cdd9517c44284386a8887b667b58c5a4"
format_stamp: "Formatted at 2026-09-28 14:43:21 on dist-test-slave-0n27"
server_key: "41e502bf6c2a186401a9b8a0ac6b97af"
server_key_iv: "8755b49ef2ed0fbc0c255aedc7546e7d"
server_key_version: "encryptionkey@0"
I20260928 14:43:21.408912 15320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:43:21.411255 15336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:21.412428 15320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:43:21.412565 15320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/wal
uuid: "cdd9517c44284386a8887b667b58c5a4"
format_stamp: "Formatted at 2026-09-28 14:43:21 on dist-test-slave-0n27"
server_key: "41e502bf6c2a186401a9b8a0ac6b97af"
server_key_iv: "8755b49ef2ed0fbc0c255aedc7546e7d"
server_key_version: "encryptionkey@0"
I20260928 14:43:21.412690 15320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:21.428944 15320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:21.432615 15320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:21.432828 15320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:21.439383 15320 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.60:37727
I20260928 14:43:21.439455 15388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.60:37727 every 8 connection(s)
I20260928 14:43:21.440357 15320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
I20260928 14:43:21.443141 15389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:21.446343 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 15320
I20260928 14:43:21.446504 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606444849692-1828-0/minicluster-data/master-2/wal/instance
I20260928 14:43:21.446835 1828 external_mini_cluster.cc:1428] Setting key 6bcf28954600324e2b83928a8641bd85
I20260928 14:43:21.446993 15389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606601, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
I20260928 14:43:21.459075 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.448979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.1.201.62:44763)
0928 14:43:21.449118 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.449122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.449404 (+ 282us) server_negotiation.cc:407] Connection header received
0928 14:43:21.449825 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.449829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.449890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.449979 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.450755 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.451287 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.452245 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.452461 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.452567 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.455092 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.455095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.455097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.455119 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.456747 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.457320 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.457324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.457326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.457411 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.457763 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.457766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.457767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.458060 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.458155 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.458774 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.458900 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:43:21.459955 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.448882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44763 (local address 127.0.0.1:60528)
0928 14:43:21.449272 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.449308 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:21.449586 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:21.450030 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:21.450036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:21.450364 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:21.450613 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.450626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.451406 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.451411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.452129 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.452137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.452609 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.452613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.452802 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.453376 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:21.453400 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:21.454936 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:21.456854 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.456863 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.456874 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.457224 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.457501 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.457503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.457505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.457666 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.458173 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:21.458178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:21.458523 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 14:43:21.458692 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 14:43:21.458997 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":94,"spinlock_wait_cycles":3200,"thread_start_us":137,"threads_started":1}
I20260928 14:43:21.459955 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.448188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.1.201.60:37727)
0928 14:43:21.448631 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.448648 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.448688 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:43:21.449524 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.449544 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.449774 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.450005 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.450821 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.451673 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.452472 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.452841 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.452949 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.455491 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.455495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.455504 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.455523 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.457173 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.457659 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.457665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.457667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.457743 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.458029 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.458031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.458032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.458523 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.458703 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.458928 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.459035 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":282,"spinlock_wait_cycles":3584,"thread_start_us":129,"threads_started":1}
I20260928 14:43:21.461107 15389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606601, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:21.469631 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.461573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:46159 (local address 127.0.0.1:41084)
0928 14:43:21.461691 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.461702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:21.461811 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:21.462076 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:21.462078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:21.462278 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:21.462396 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.462403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.463124 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.463128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.463644 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.463651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.464088 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.464091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.464190 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.464884 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:21.464894 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:21.466284 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:21.468216 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.468219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.468223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.468420 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.468701 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.468702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.468704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.468769 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.469150 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:21.469153 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:21.469254 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 14:43:21.469358 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:43:21.469481 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:43:21.469656 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.461684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.1.201.61:46159)
0928 14:43:21.461825 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.461829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.461862 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:43:21.461907 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.461910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.461958 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.462056 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.462519 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.463012 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.463755 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.463983 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.464088 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.466417 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.466420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.466423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.466449 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.468073 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.468513 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.468516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.468518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.468602 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.468849 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.468851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.468853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.469057 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.469159 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.469422 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.469521 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:43:21.470497 15389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:43:21.470562 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.462349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.1.201.62:44763)
0928 14:43:21.462455 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.462460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.462472 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:43:21.462607 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.462610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.462647 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.462733 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.463282 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.463754 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.464522 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.464715 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.464787 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.467364 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.467368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.467371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.467409 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.468873 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.469242 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.469246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.469248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.469329 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.469580 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.469584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.469586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.469913 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.470134 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.470314 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.470419 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606601, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
I20260928 14:43:21.476712 15314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } attempt: 3
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
I20260928 14:43:21.479578 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.470966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50926)
0928 14:43:21.471050 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.471062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:21.471131 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:21.471441 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:21.471443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:21.471603 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:21.471688 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.471697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.472718 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.472721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.473507 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.473515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.474065 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.474068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.474138 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.474519 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:21.474532 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:21.475736 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:21.478024 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.478028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.478030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.478277 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.478571 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.478576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.478578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.478663 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.479082 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:21.479084 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:21.479164 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 14:43:21.479263 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:43:21.479390 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":1}
I20260928 14:43:21.480176 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.471168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.1.201.60:37727)
0928 14:43:21.471261 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.471270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.471279 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:43:21.471301 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.471303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.471329 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.471431 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.471850 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.472624 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.473612 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.473971 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.474066 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.475866 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.475871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.475875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.475904 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.477939 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.478393 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.478399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.478402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.478474 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.478758 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.478763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.478765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.478974 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.479793 (+ 819us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.479881 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.479990 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":39}
I20260928 14:43:21.481901 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.471651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.1.201.61:46159)
0928 14:43:21.471802 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.471806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.471820 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:43:21.471910 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.471912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.471943 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.471996 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.472631 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.473371 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.474384 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.474719 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.474822 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.477507 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.477510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.477513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.477538 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.479516 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.480231 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.480235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.480236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.480521 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.480827 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.480831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.480832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.481086 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.481322 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.481466 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.481571 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:43:21.481882 15238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } attempt: 3
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606601, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
I20260928 14:43:21.483729 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4: Bootstrap starting.
I20260928 14:43:21.485294 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:43:21.485960 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.477097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50932)
0928 14:43:21.477214 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.477229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:21.477309 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:21.477733 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:21.477736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:21.477953 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:21.478065 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.478076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.479272 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.479276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.479886 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.479897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.480426 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.480429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.480518 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.481037 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:21.481108 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:21.482427 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:21.484376 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.484379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.484381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.484607 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.484927 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.484930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.484932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.485018 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.485465 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:21.485468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:21.485551 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 14:43:21.485621 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:43:21.485750 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"spinlock_wait_cycles":23808}
I20260928 14:43:21.486547 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.477173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.1.201.60:37727)
0928 14:43:21.477543 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.477550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.477560 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:43:21.477585 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.477587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.477617 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.477716 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.478170 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.479148 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.480005 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.480304 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.480748 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.482563 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.482566 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.482568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.482592 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.484239 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.484749 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.484752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.484753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.484827 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.485126 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.485130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.485131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.485356 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.486198 (+ 842us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.486275 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.486380 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
I20260928 14:43:21.486655 15389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:21.487632 15389 log.cc:824] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:21.489192 15314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d: Bootstrap starting.
I20260928 14:43:21.489500 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4: No bootstrap required, opened a new log
Sep 28 14:43:21 dist-test-slave-0n27 krb5kdc[15157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606600, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
I20260928 14:43:21.491827 15314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:21.492695 15314 log.cc:824] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:21.494266 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.482247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:37727 (local address 127.0.0.1:50948)
0928 14:43:21.482488 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:21.482504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:21.482569 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:21.483311 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:21.483316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:21.483755 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:21.483873 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.483882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.484932 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.484935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.485720 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:21.485731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.486331 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.486336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:21.486447 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.486896 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:21.486914 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:21.490755 (+ 3841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:21.492621 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.492624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.492626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.492857 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.493205 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:21.493208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:21.493210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:21.493301 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:21.493717 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:21.493720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:21.493833 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 14:43:21.493942 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:43:21.494076 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260928 14:43:21.494293 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.482690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.1.201.60:37727)
0928 14:43:21.482795 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.482801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.483074 (+ 273us) server_negotiation.cc:407] Connection header received
0928 14:43:21.483115 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.483117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.483172 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.483988 (+ 816us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.484079 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.484816 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.485892 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.486208 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.487529 (+ 1321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.490887 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.490891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.490894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.490918 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.492497 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.493002 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.493005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.493006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.493079 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.493419 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.493422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.493424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.493608 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.493699 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.494003 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.494129 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:43:21.494210 15389 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } }
I20260928 14:43:21.494462 15389 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:21.494522 15389 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd9517c44284386a8887b667b58c5a4, State: Initialized, Role: FOLLOWER
I20260928 14:43:21.494855 15314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d: No bootstrap required, opened a new log
I20260928 14:43:21.494998 15389 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } }
I20260928 14:43:21.495465 15404 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } } }
I20260928 14:43:21.495659 15404 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:43:21.496228 15389 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:43:21.497120 15238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87: Bootstrap starting.
I20260928 14:43:21.498795 15314 raft_consensus.cc:348] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } }
I20260928 14:43:21.499749 15238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:21.500846 15238 log.cc:824] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:21.500000 15320 master_runner.cc:186] Error getting master registration for 127.1.201.62:44763: OK, instance_id { permanent_uuid: "2c37907173234392852ef9554f7bfc87" instance_seqno: 1790606601019382 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:43:21.503479 15320 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:43:21.503537 15314 raft_consensus.cc:374] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:21.503597 15314 raft_consensus.cc:729] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74402dfc46a45e7910bf6e205c4a79d, State: Initialized, Role: FOLLOWER
I20260928 14:43:21.504087 15314 consensus_queue.cc:260] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } }
I20260928 14:43:21.505430 15314 sys_catalog.cc:564] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:43:21.507269 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.491656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.1.201.61:46159)
0928 14:43:21.491795 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.491799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.498144 (+ 6345us) server_negotiation.cc:407] Connection header received
0928 14:43:21.498349 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.498352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.498411 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.498465 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.499606 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.500327 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.501469 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.501849 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.501928 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.503621 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.503627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.503632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.503653 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.505287 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.505766 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.505800 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.505802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.505889 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.506263 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.506268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.506270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.506492 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.506611 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.506988 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.507106 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:43:21.508980 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.491258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.1.201.62:44763)
0928 14:43:21.491365 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.491369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.500044 (+ 8675us) server_negotiation.cc:407] Connection header received
0928 14:43:21.500254 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.500257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.500295 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.500341 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.501273 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.502006 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.503045 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.503439 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.503524 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.505296 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.505299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.505302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.505331 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.506999 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.507653 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.507658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.507659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.507747 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.508061 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.508064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.508066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.508278 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.508373 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.508724 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.508847 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:43:21.511394 15238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87: No bootstrap required, opened a new log
W20260928 14:43:21.512123 15428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:43:21.512200 15428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 14:43:21.512167 15415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:43:21.512352 15415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:43:21.509994 15245 master_runner.cc:186] Error getting master registration for 127.1.201.62:44763: OK, instance_id { permanent_uuid: "2c37907173234392852ef9554f7bfc87" instance_seqno: 1790606601019382 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:43:21.513831 15245 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:43:21.508188 15417 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } } }
I20260928 14:43:21.513929 15417 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:43:21.515496 15238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } }
I20260928 14:43:21.515734 15238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:21.515787 15238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c37907173234392852ef9554f7bfc87, State: Initialized, Role: FOLLOWER
I20260928 14:43:21.516820 15238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } }
I20260928 14:43:21.518139 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:21.492132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.1.201.60:37727)
0928 14:43:21.492250 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:21.492253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:21.499779 (+ 7526us) server_negotiation.cc:407] Connection header received
0928 14:43:21.499878 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:21.499881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:21.499922 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:21.499994 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:21.502046 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.503077 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.504777 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:21.505111 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:21.505198 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:21.507336 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:21.507342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:21.507348 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:21.507377 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:21.509128 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.516430 (+ 7302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.516434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.516436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.516536 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:21.516863 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:21.516866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:21.516868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:21.517285 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:21.517425 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:21.517846 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 14:43:21.517988 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:43:21.518380 15238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:43:21.524499 15443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:43:21.524636 15443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:43:21.529386 15170 master_runner.cc:263] No leader master found from master 2c37907173234392852ef9554f7bfc87
I20260928 14:43:21.529454 15170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:43:21.529970 15431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } } }
I20260928 14:43:21.530110 15431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:43:21.534202 15431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:43:21.534344 15431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } }
I20260928 14:43:21.535333 15431 leader_election.cc:288] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74402dfc46a45e7910bf6e205c4a79d (127.1.201.61:46159), cdd9517c44284386a8887b667b58c5a4 (127.1.201.60:37727)
I20260928 14:43:21.535809 15287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c37907173234392852ef9554f7bfc87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74402dfc46a45e7910bf6e205c4a79d" is_pre_election: true
I20260928 14:43:21.535874 15363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c37907173234392852ef9554f7bfc87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd9517c44284386a8887b667b58c5a4" is_pre_election: true
I20260928 14:43:21.536080 15287 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c37907173234392852ef9554f7bfc87 in term 0.
I20260928 14:43:21.536166 15363 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c37907173234392852ef9554f7bfc87 in term 0.
I20260928 14:43:21.536597 15188 leader_election.cc:302] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c37907173234392852ef9554f7bfc87, a74402dfc46a45e7910bf6e205c4a79d, cdd9517c44284386a8887b667b58c5a4; no voters:
I20260928 14:43:21.537019 15431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:43:21.537102 15431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:43:21.537159 15431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:21.538513 15431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } }
I20260928 14:43:21.539252 15287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c37907173234392852ef9554f7bfc87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74402dfc46a45e7910bf6e205c4a79d"
I20260928 14:43:21.539373 15287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:21.539408 15363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c37907173234392852ef9554f7bfc87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd9517c44284386a8887b667b58c5a4"
I20260928 14:43:21.539520 15363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:21.540616 15287 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c37907173234392852ef9554f7bfc87 in term 1.
I20260928 14:43:21.540782 15363 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c37907173234392852ef9554f7bfc87 in term 1.
I20260928 14:43:21.540993 15189 leader_election.cc:302] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c37907173234392852ef9554f7bfc87, a74402dfc46a45e7910bf6e205c4a79d; no voters:
I20260928 14:43:21.541110 15431 leader_election.cc:288] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [CANDIDATE]: Term 1 election: Requested vote from peers a74402dfc46a45e7910bf6e205c4a79d (127.1.201.61:46159), cdd9517c44284386a8887b667b58c5a4 (127.1.201.60:37727)
I20260928 14:43:21.541203 15431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:43:21.541265 15431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [term 1 LEADER]: Becoming Leader. State: Replica: 2c37907173234392852ef9554f7bfc87, State: Running, Role: LEADER
I20260928 14:43:21.541571 15431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } }
I20260928 14:43:21.543378 15431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c37907173234392852ef9554f7bfc87. Latest consensus state: current_term: 1 leader_uuid: "2c37907173234392852ef9554f7bfc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } } }
I20260928 14:43:21.543484 15431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:21.550123 15447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:43:21.551151 15447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:43:21.559108 15363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [term 1 FOLLOWER]: Refusing update from remote peer 2c37907173234392852ef9554f7bfc87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:43:21.559942 15287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [term 1 FOLLOWER]: Refusing update from remote peer 2c37907173234392852ef9554f7bfc87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:43:21.560580 15445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:43:21.560747 15431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:43:21.564461 15417 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c37907173234392852ef9554f7bfc87. Latest consensus state: current_term: 1 leader_uuid: "2c37907173234392852ef9554f7bfc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } } }
I20260928 14:43:21.564553 15417 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:43:21.568447 15404 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c37907173234392852ef9554f7bfc87. Latest consensus state: current_term: 1 leader_uuid: "2c37907173234392852ef9554f7bfc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } } }
I20260928 14:43:21.568571 15404 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:43:21.571123 15445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c37907173234392852ef9554f7bfc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } } }
I20260928 14:43:21.571241 15445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:21.572212 15404 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c37907173234392852ef9554f7bfc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } } }
I20260928 14:43:21.572321 15404 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:43:21.572602 15448 mvcc.cc:205] Tried to move back new op lower bound from 7334324639978057728 to 7334324639917568000. Current Snapshot: MvccSnapshot[applied={T|T < 7334324639978057728}]
I20260928 14:43:21.575758 15451 mvcc.cc:205] Tried to move back new op lower bound from 7334324639978057728 to 7334324639917568000. Current Snapshot: MvccSnapshot[applied={T|T < 7334324639978057728}]
I20260928 14:43:21.577000 15431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c37907173234392852ef9554f7bfc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } } }
I20260928 14:43:21.577109 15431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87 [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:21.577569 15447 catalog_manager.cc:1347] Generated new cluster ID: 0622a331ee4248e4b450a82b8a34ad9e
I20260928 14:43:21.577620 15447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:43:21.578087 15417 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c37907173234392852ef9554f7bfc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c37907173234392852ef9554f7bfc87" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44763 } } peers { permanent_uuid: "a74402dfc46a45e7910bf6e205c4a79d" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 46159 } } peers { permanent_uuid: "cdd9517c44284386a8887b667b58c5a4" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 37727 } } }
I20260928 14:43:21.578190 15417 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:43:21.578850 15449 mvcc.cc:205] Tried to move back new op lower bound from 7334324639978057728 to 7334324639917568000. Current Snapshot: MvccSnapshot[applied={T|T < 7334324639978057728}]
I20260928 14:43:21.592126 15447 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:43:21.593376 15447 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:43:21.608713 15447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c37907173234392852ef9554f7bfc87: Generated new TSK 0
I20260928 14:43:21.609598 15447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:43:22.513458 15428 catalog_manager.cc:1259] Loaded cluster ID: 0622a331ee4248e4b450a82b8a34ad9e
I20260928 14:43:22.513543 15428 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d: loading cluster ID for follower catalog manager: success
I20260928 14:43:22.513489 15415 catalog_manager.cc:1259] Loaded cluster ID: 0622a331ee4248e4b450a82b8a34ad9e
I20260928 14:43:22.513618 15415 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4: loading cluster ID for follower catalog manager: success
I20260928 14:43:22.515369 15415 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4: acquiring CA information for follower catalog manager: success
I20260928 14:43:22.515369 15428 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d: acquiring CA information for follower catalog manager: success
I20260928 14:43:22.516651 15415 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cdd9517c44284386a8887b667b58c5a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:43:22.516650 15428 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a74402dfc46a45e7910bf6e205c4a79d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:43:22.674085 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 15170
I20260928 14:43:22.680411 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 15245
I20260928 14:43:22.686043 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 15320
2026-09-28T14:43:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3014 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:43:22 dist-test-slave-0n27 krb5kdc[15463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:43:22 dist-test-slave-0n27 krb5kdc[15463](info): set up 2 sockets
Sep 28 14:43:22 dist-test-slave-0n27 krb5kdc[15463](info): commencing operation
krb5kdc: starting...
W20260928 14:43:23.722860 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:43:23 dist-test-slave-0n27 krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:43:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:43:23.842131 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:45361
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:45361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:23.953469 15476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:23.953727 15476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:23.953766 15476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:23.957847 15476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:43:23.957911 15476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:23.957929 15476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:23.957944 15476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:43:23.957958 15476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:43:23.963336 15476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:45361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:23.964408 15476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:23.965474 15476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:23.974267 15484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:23.974228 15481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:23.974275 15476 server_base.cc:1034] running on GCE node
W20260928 14:43:23.974228 15482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:23.975045 15476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:23.976168 15476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:23.977324 15476 hybrid_clock.cc:648] HybridClock initialized: now 1790606603977294 us; error 45 us; skew 500 ppm
Sep 28 14:43:23 dist-test-slave-0n27 krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606603, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:23.980014 15476 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:43:23.980854 15476 webserver.cc:460] Webserver started at http://127.1.201.62:44137/ using document root <none> and password file <none>
I20260928 14:43:23.981374 15476 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:23.981428 15476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:23.981594 15476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:23.983685 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "ae2e441ef5844daa8d9c9bde7de0e227"
format_stamp: "Formatted at 2026-09-28 14:43:23 on dist-test-slave-0n27"
server_key: "6670caa5b35e0370c4fa05d0dd17830e"
server_key_iv: "961ea5dee446f0f37379851e659973f8"
server_key_version: "encryptionkey@0"
I20260928 14:43:23.984184 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "ae2e441ef5844daa8d9c9bde7de0e227"
format_stamp: "Formatted at 2026-09-28 14:43:23 on dist-test-slave-0n27"
server_key: "6670caa5b35e0370c4fa05d0dd17830e"
server_key_iv: "961ea5dee446f0f37379851e659973f8"
server_key_version: "encryptionkey@0"
I20260928 14:43:23.987751 15476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:43:23.990166 15491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:23.991396 15476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:43:23.991665 15476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "ae2e441ef5844daa8d9c9bde7de0e227"
format_stamp: "Formatted at 2026-09-28 14:43:23 on dist-test-slave-0n27"
server_key: "6670caa5b35e0370c4fa05d0dd17830e"
server_key_iv: "961ea5dee446f0f37379851e659973f8"
server_key_version: "encryptionkey@0"
I20260928 14:43:23.991765 15476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:24.012933 15476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:24.016296 15476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:24.016450 15476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:24.023497 15476 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:45361
I20260928 14:43:24.023566 15543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:45361 every 8 connection(s)
I20260928 14:43:24.024473 15476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:43:24.027487 15544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:24.028937 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 15476
I20260928 14:43:24.029085 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:43:24.029438 1828 external_mini_cluster.cc:1428] Setting key 4c5ae08f9974295aeed02ffaf73da924
I20260928 14:43:24.033536 15544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227: Bootstrap starting.
I20260928 14:43:24.035790 15544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:24.036718 15544 log.cc:824] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:43:24 dist-test-slave-0n27 krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:24.038789 15544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227: No bootstrap required, opened a new log
I20260928 14:43:24.042456 15544 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2e441ef5844daa8d9c9bde7de0e227" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45361 } }
I20260928 14:43:24.042794 15544 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:24.042876 15544 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2e441ef5844daa8d9c9bde7de0e227, State: Initialized, Role: FOLLOWER
I20260928 14:43:24.043349 15544 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2e441ef5844daa8d9c9bde7de0e227" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45361 } }
I20260928 14:43:24.043512 15544 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:43:24.043521 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:24.030893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.1.201.62:45361)
0928 14:43:24.031342 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:24.031357 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:24.031393 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:43:24.031988 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:24.032003 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:24.032331 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:24.032662 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:24.033463 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:24.034532 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:24.035419 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:24.035932 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:24.036079 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:24.038801 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:24.038806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:24.038818 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:24.038843 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:24.040568 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:24.041063 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:24.041074 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:24.041079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:24.041201 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:24.041497 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:24.041501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:24.041503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:24.041959 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:24.042176 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:24.042398 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 14:43:24.042686 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":150,"threads_started":1}
I20260928 14:43:24.043619 15544 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:43:24.043726 15544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:24.044818 15544 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2e441ef5844daa8d9c9bde7de0e227" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45361 } }
I20260928 14:43:24.045161 15544 leader_election.cc:302] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae2e441ef5844daa8d9c9bde7de0e227; no voters:
I20260928 14:43:24.045499 15544 leader_election.cc:288] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:43:24.045615 15549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:43:24.045855 15549 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [term 1 LEADER]: Becoming Leader. State: Replica: ae2e441ef5844daa8d9c9bde7de0e227, State: Running, Role: LEADER
I20260928 14:43:24.046240 15549 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2e441ef5844daa8d9c9bde7de0e227" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45361 } }
I20260928 14:43:24.046710 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:43:24.046839 15544 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:43:24.047842 15550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae2e441ef5844daa8d9c9bde7de0e227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2e441ef5844daa8d9c9bde7de0e227" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45361 } } }
I20260928 14:43:24.047986 15550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:24.048221 15550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae2e441ef5844daa8d9c9bde7de0e227. Latest consensus state: current_term: 1 leader_uuid: "ae2e441ef5844daa8d9c9bde7de0e227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2e441ef5844daa8d9c9bde7de0e227" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 45361 } } }
I20260928 14:43:24.048311 15550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227 [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:24.049211 15558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:43:24.051934 15558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:43:24.057330 15558 catalog_manager.cc:1347] Generated new cluster ID: 8bda5ebcac2841ec826f907f3a196e99
I20260928 14:43:24.057399 15558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:43:24.096937 15558 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:43:24.097991 15558 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:43:24.109748 15558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae2e441ef5844daa8d9c9bde7de0e227: Generated new TSK 0
I20260928 14:43:24.110633 15558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:43:24.154175 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 15476
2026-09-28T14:43:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1467 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:43:24 dist-test-slave-0n27 krb5kdc[15571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:43:24 dist-test-slave-0n27 krb5kdc[15571](info): set up 2 sockets
Sep 28 14:43:24 dist-test-slave-0n27 krb5kdc[15571](info): commencing operation
krb5kdc: starting...
W20260928 14:43:25.194944 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:43:25 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:43:25.317094 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42193
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:42193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:25.440709 15584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:25.440966 15584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:25.441005 15584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:25.444950 15584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:43:25.445007 15584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:25.445026 15584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:25.445042 15584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:43:25.445057 15584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:43:25.450018 15584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:42193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:25.451001 15584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:25.451948 15584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:25.460309 15590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:25.460330 15589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:25.460376 15584 server_base.cc:1034] running on GCE node
W20260928 14:43:25.460320 15592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:25.461043 15584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:25.462168 15584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:25.463330 15584 hybrid_clock.cc:648] HybridClock initialized: now 1790606605463313 us; error 30 us; skew 500 ppm
Sep 28 14:43:25 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606605, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:25.466009 15584 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:43:25.466822 15584 webserver.cc:460] Webserver started at http://127.1.201.62:38177/ using document root <none> and password file <none>
I20260928 14:43:25.467316 15584 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:25.467371 15584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:25.467530 15584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:25.469441 15584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "3303ce6b76224d5897a4710bb5dbc2ba"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "869fa5c74399849db664a5df9e384b0b"
server_key_iv: "d89b308a2eef9bd90cb8c7da6246ccc9"
server_key_version: "encryptionkey@0"
I20260928 14:43:25.470005 15584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "3303ce6b76224d5897a4710bb5dbc2ba"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "869fa5c74399849db664a5df9e384b0b"
server_key_iv: "d89b308a2eef9bd90cb8c7da6246ccc9"
server_key_version: "encryptionkey@0"
I20260928 14:43:25.473630 15584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:43:25.475999 15599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:25.477187 15584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:43:25.477316 15584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "3303ce6b76224d5897a4710bb5dbc2ba"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "869fa5c74399849db664a5df9e384b0b"
server_key_iv: "d89b308a2eef9bd90cb8c7da6246ccc9"
server_key_version: "encryptionkey@0"
I20260928 14:43:25.477416 15584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:25.489468 15584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:25.492339 15584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:25.492537 15584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:25.499392 15584 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:42193
I20260928 14:43:25.499648 15651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:42193 every 8 connection(s)
I20260928 14:43:25.500366 15584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:43:25.503150 15652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:25.504698 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 15584
I20260928 14:43:25.504828 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:43:25.505105 1828 external_mini_cluster.cc:1428] Setting key acb58fed69b3aeb79c4e8ff5b4126121
I20260928 14:43:25.509186 15652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Bootstrap starting.
I20260928 14:43:25.511787 15652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:25.512697 15652 log.cc:824] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:43:25 dist-test-slave-0n27 krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:25.514705 15652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: No bootstrap required, opened a new log
I20260928 14:43:25.518306 15652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:25.518558 15652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:25.518635 15652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3303ce6b76224d5897a4710bb5dbc2ba, State: Initialized, Role: FOLLOWER
I20260928 14:43:25.518983 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:25.506414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.1.201.62:42193)
0928 14:43:25.506812 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:25.506828 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:25.506863 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:43:25.507463 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:25.507479 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:25.507778 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:25.508052 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:25.508904 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:25.509954 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:25.511132 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:25.511591 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:25.511715 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:25.514488 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:25.514493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:25.514505 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:25.514530 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:25.516090 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:25.516535 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:25.516546 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:25.516550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:25.516653 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:25.516960 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:25.516964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:25.516965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:25.517343 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:25.517624 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:25.517886 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 14:43:25.518165 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":125,"threads_started":1}
I20260928 14:43:25.519166 15652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:25.519335 15652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:43:25.519402 15652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:43:25.519500 15652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:25.520398 15652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:25.520745 15652 leader_election.cc:302] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3303ce6b76224d5897a4710bb5dbc2ba; no voters:
I20260928 14:43:25.521070 15652 leader_election.cc:288] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:43:25.521145 15657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:43:25.521392 15657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 1 LEADER]: Becoming Leader. State: Replica: 3303ce6b76224d5897a4710bb5dbc2ba, State: Running, Role: LEADER
I20260928 14:43:25.521678 15657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:25.522398 15652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:43:25.523536 15658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } } }
I20260928 14:43:25.523658 15658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:25.524056 15666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:43:25.524564 15659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3303ce6b76224d5897a4710bb5dbc2ba. Latest consensus state: current_term: 1 leader_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } } }
I20260928 14:43:25.524742 15659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:25.527235 15666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:43:25.532392 15666 catalog_manager.cc:1347] Generated new cluster ID: 551865a3e4294f81aeed8a0f7f33d02d
I20260928 14:43:25.532451 15666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:43:25.542765 15666 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:43:25.543853 15666 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:43:25.548056 15666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Generated new TSK 0
I20260928 14:43:25.548656 15666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:43:25.578855 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42193
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:25.688283 15678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:25.688519 15678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:25.688588 15678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:25.692592 15678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:25.692646 15678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:25.692752 15678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:43:25.697991 15678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:25.698992 15678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:25.700004 15678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:25.709689 15684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:25.709707 15678 server_base.cc:1034] running on GCE node
W20260928 14:43:25.709844 15686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:25.709683 15683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:25.710258 15678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:25.710826 15678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:25.712018 15678 hybrid_clock.cc:648] HybridClock initialized: now 1790606605712004 us; error 26 us; skew 500 ppm
Sep 28 14:43:25 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606605, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:25.714708 15678 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:43:25.715508 15678 webserver.cc:460] Webserver started at http://127.1.201.1:43823/ using document root <none> and password file <none>
I20260928 14:43:25.716068 15678 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:25.716130 15678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:25.716367 15678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:25.718449 15678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "7af2e4d277844e1aaf9e918e942e3b68"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "7b2fd83946c200d1115a298f7f51b5c9"
server_key_iv: "4a78b411c72998635fecc14c0a944c7a"
server_key_version: "encryptionkey@0"
I20260928 14:43:25.718999 15678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "7af2e4d277844e1aaf9e918e942e3b68"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "7b2fd83946c200d1115a298f7f51b5c9"
server_key_iv: "4a78b411c72998635fecc14c0a944c7a"
server_key_version: "encryptionkey@0"
I20260928 14:43:25.722738 15678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:43:25.725124 15693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:25.726370 15678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:43:25.726493 15678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "7af2e4d277844e1aaf9e918e942e3b68"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "7b2fd83946c200d1115a298f7f51b5c9"
server_key_iv: "4a78b411c72998635fecc14c0a944c7a"
server_key_version: "encryptionkey@0"
I20260928 14:43:25.726590 15678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:25.753464 15678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:25.756381 15678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:25.756584 15678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:25.757179 15678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:25.758061 15678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:25.758126 15678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:25.758191 15678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:25.758255 15678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:25.767012 15678 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:35723
I20260928 14:43:25.767041 15807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:35723 every 8 connection(s)
I20260928 14:43:25.768079 15678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:43:25.775329 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 15678
I20260928 14:43:25.775466 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:43:25.775842 1828 external_mini_cluster.cc:1428] Setting key 5105f2136ce82afb3b7003a5557b9fe3
Sep 28 14:43:25 dist-test-slave-0n27 krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606605, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:25.783000 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:25.770084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:45629 (local address 127.1.201.62:42193)
0928 14:43:25.770231 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:25.770234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:25.771177 (+ 943us) server_negotiation.cc:407] Connection header received
0928 14:43:25.772393 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:25.772396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:25.772443 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:25.772516 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:25.773647 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:25.774236 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:25.775335 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:25.775720 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:25.775877 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:25.778925 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:25.778927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:25.778929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:25.778952 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:25.780641 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:25.781136 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:25.781142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:25.781147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:25.781241 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:25.781651 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:25.781657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:25.781676 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:25.782015 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:25.782129 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:25.782628 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 14:43:25.782778 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:43:25.783697 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:25.770560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:45629)
0928 14:43:25.771022 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:25.771066 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:25.772199 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:25.772646 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:25.772652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:25.772990 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:25.773519 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:25.773533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:25.774354 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:25.774356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:25.775219 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:25.775228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:25.775933 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:25.775941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:25.776092 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:25.776761 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:25.776781 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:25.778752 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:25.780774 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:25.780779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:25.780790 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:25.781035 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:25.781369 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:25.781375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:25.781380 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:25.781563 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:25.782140 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:25.782148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:25.782388 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 14:43:25.782544 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 14:43:25.782787 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":130,"threads_started":1}
I20260928 14:43:25.785017 15808 heartbeater.cc:347] Connected to a master server at 127.1.201.62:42193
I20260928 14:43:25.785259 15808 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:25.785728 15808 heartbeater.cc:511] Master 127.1.201.62:42193 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
I20260928 14:43:25.787318 15616 ts_manager.cc:194] Registered new tserver with Master: 7af2e4d277844e1aaf9e918e942e3b68 (127.1.201.1:35723)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
I20260928 14:43:25.788554 15616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:45629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:43:25.804071 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42193
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:25.913863 15814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:25.914083 15814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:25.914119 15814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:25.918056 15814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:25.918103 15814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:25.918198 15814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:43:25.923388 15814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:25.924458 15814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:25.925446 15814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:25.934788 15822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:25.934798 15820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:25.934805 15819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:25.934883 15814 server_base.cc:1034] running on GCE node
I20260928 14:43:25.935492 15814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:25.936101 15814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:25.937270 15814 hybrid_clock.cc:648] HybridClock initialized: now 1790606605937248 us; error 30 us; skew 500 ppm
Sep 28 14:43:25 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606605, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:25.940001 15814 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:43:25.940834 15814 webserver.cc:460] Webserver started at http://127.1.201.2:33687/ using document root <none> and password file <none>
I20260928 14:43:25.941395 15814 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:25.941484 15814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:25.941706 15814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:25.943728 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "0a99558758e243a58f97846f55e226ea"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "5ec637d7547dac9eda14c00f23ac6ca7"
server_key_iv: "ea016a3e8dc49c60e4f1cae0973e311d"
server_key_version: "encryptionkey@0"
I20260928 14:43:25.944314 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "0a99558758e243a58f97846f55e226ea"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "5ec637d7547dac9eda14c00f23ac6ca7"
server_key_iv: "ea016a3e8dc49c60e4f1cae0973e311d"
server_key_version: "encryptionkey@0"
I20260928 14:43:25.947896 15814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 14:43:25.950191 15829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:25.951371 15814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:43:25.951520 15814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "0a99558758e243a58f97846f55e226ea"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "5ec637d7547dac9eda14c00f23ac6ca7"
server_key_iv: "ea016a3e8dc49c60e4f1cae0973e311d"
server_key_version: "encryptionkey@0"
I20260928 14:43:25.951627 15814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:25.969992 15814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:25.972780 15814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:25.973013 15814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:25.973583 15814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:25.974462 15814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:25.974509 15814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:25.974546 15814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:25.974561 15814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:25.983572 15814 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:39559
I20260928 14:43:25.983597 15943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:39559 every 8 connection(s)
I20260928 14:43:25.984443 15814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:43:25.993384 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 15814
I20260928 14:43:25.993534 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
Sep 28 14:43:25 dist-test-slave-0n27 krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606605, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:25.993888 1828 external_mini_cluster.cc:1428] Setting key 74ec1dfd7e5786b4f03eea250986468d
I20260928 14:43:25.998080 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:25.986245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:46453 (local address 127.1.201.62:42193)
0928 14:43:25.986377 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:25.986380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:25.987149 (+ 769us) server_negotiation.cc:407] Connection header received
0928 14:43:25.988440 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:25.988443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:25.988502 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:25.988620 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:25.989728 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:25.990325 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:25.991472 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:25.991689 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:25.991795 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:25.994334 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:25.994339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:25.994342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:25.994408 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:25.995942 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:25.996557 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:25.996559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:25.996561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:25.996631 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:25.997026 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:25.997028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:25.997029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:25.997216 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:25.997306 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:25.997804 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 14:43:25.997918 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:43:25.998831 15946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:25.986570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:46453)
0928 14:43:25.986980 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:25.987039 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:25.988225 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:25.988788 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:25.988795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:25.989128 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:25.989575 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:25.989588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:25.990451 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:25.990456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:25.991335 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:25.991344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:25.991829 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:25.991832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:25.991996 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:25.992620 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:25.992635 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:25.994166 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:25.996062 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:25.996070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:25.996086 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:25.996458 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:25.996723 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:25.996729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:25.996734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:25.996937 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:25.997308 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:25.997317 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:25.997545 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 14:43:25.997731 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 14:43:25.998050 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
I20260928 14:43:25.999989 15944 heartbeater.cc:347] Connected to a master server at 127.1.201.62:42193
I20260928 14:43:26.000257 15944 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:26.000730 15944 heartbeater.cc:511] Master 127.1.201.62:42193 requested a full tablet report, sending...
I20260928 14:43:26.001859 15616 ts_manager.cc:194] Registered new tserver with Master: 0a99558758e243a58f97846f55e226ea (127.1.201.2:39559)
I20260928 14:43:26.002663 15616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:46453
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:43:26.021297 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42193
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:26.130554 15949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:26.130867 15949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:26.130931 15949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:26.134936 15949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:26.134994 15949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:26.135051 15949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:43:26.140087 15949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:26.141014 15949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:26.142024 15949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:26.151319 15954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:26.151360 15949 server_base.cc:1034] running on GCE node
W20260928 14:43:26.151304 15955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:26.151306 15957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:26.151942 15949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:26.152489 15949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:26.153644 15949 hybrid_clock.cc:648] HybridClock initialized: now 1790606606153621 us; error 32 us; skew 500 ppm
Sep 28 14:43:26 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:26.156486 15949 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:43:26.157325 15949 webserver.cc:460] Webserver started at http://127.1.201.3:32921/ using document root <none> and password file <none>
I20260928 14:43:26.157924 15949 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:26.157989 15949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:26.158229 15949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:26.160265 15949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0"
format_stamp: "Formatted at 2026-09-28 14:43:26 on dist-test-slave-0n27"
server_key: "69a02bea76c5fdd01eb4cfa209f492f0"
server_key_iv: "f0b2a7e0d5398379f243fed83aaf2cd8"
server_key_version: "encryptionkey@0"
I20260928 14:43:26.160830 15949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0"
format_stamp: "Formatted at 2026-09-28 14:43:26 on dist-test-slave-0n27"
server_key: "69a02bea76c5fdd01eb4cfa209f492f0"
server_key_iv: "f0b2a7e0d5398379f243fed83aaf2cd8"
server_key_version: "encryptionkey@0"
I20260928 14:43:26.164424 15949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:43:26.166805 15964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:26.167984 15949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:43:26.168113 15949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0"
format_stamp: "Formatted at 2026-09-28 14:43:26 on dist-test-slave-0n27"
server_key: "69a02bea76c5fdd01eb4cfa209f492f0"
server_key_iv: "f0b2a7e0d5398379f243fed83aaf2cd8"
server_key_version: "encryptionkey@0"
I20260928 14:43:26.168213 15949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:26.188082 15949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:26.190981 15949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:26.191186 15949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:26.191784 15949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:26.192605 15949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:26.192648 15949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:26.192746 15949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:26.192780 15949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:26.201762 15949 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:44189
I20260928 14:43:26.201845 16078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:44189 every 8 connection(s)
I20260928 14:43:26.202713 15949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:43:26.207866 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 15949
I20260928 14:43:26.208012 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:43:26.208287 1828 external_mini_cluster.cc:1428] Setting key 438a01c05cefd7fa349ee58823deb8da
Sep 28 14:43:26 dist-test-slave-0n27 krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:26.215967 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.204522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:44955 (local address 127.1.201.62:42193)
0928 14:43:26.204674 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.204679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.205513 (+ 834us) server_negotiation.cc:407] Connection header received
0928 14:43:26.206705 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.206707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.206751 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.206824 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:26.207939 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.208624 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.209928 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.210248 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.210353 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.212698 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:26.212701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:26.212703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:26.212726 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:26.214279 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:26.214702 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:26.214705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:26.214706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:26.214779 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:26.215061 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:26.215063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:26.215064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:26.215224 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:26.215307 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.215754 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.215850 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:43:26.216744 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.204941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:44955)
0928 14:43:26.205365 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.205406 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:26.206520 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:26.206979 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:26.206985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:26.207321 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:26.207785 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.207798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.208758 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.208761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.209762 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.209771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.210407 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.210411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.210595 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.211137 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:26.211153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:26.212515 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:26.214371 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:26.214375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:26.214385 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:26.214618 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:26.214858 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:26.214860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:26.214862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:26.214983 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:26.215319 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:26.215324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:26.215537 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 14:43:26.215704 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 14:43:26.215945 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260928 14:43:26.217895 16079 heartbeater.cc:347] Connected to a master server at 127.1.201.62:42193
I20260928 14:43:26.218106 16079 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:26.218526 16079 heartbeater.cc:511] Master 127.1.201.62:42193 requested a full tablet report, sending...
I20260928 14:43:26.219480 15616 ts_manager.cc:194] Registered new tserver with Master: ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189)
I20260928 14:43:26.220132 15616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:44955
I20260928 14:43:26.222239 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:43:26.231235 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.224264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.1.201.62:42193)
0928 14:43:26.224404 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.224408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.224568 (+ 160us) server_negotiation.cc:407] Connection header received
0928 14:43:26.224688 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.224691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.224747 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.224832 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:26.225457 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.226007 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.226819 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.227052 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.227164 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.228493 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:26.228496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:26.228499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:26.228530 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:26.229661 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:26.230114 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:26.230116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:26.230117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:26.230217 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:26.230486 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:26.230488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:26.230490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:26.230700 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:26.230802 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.230989 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.231078 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:43:26.233826 15616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:43:26.236730 15616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:43:26.250376 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.244646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:44189 (local address 127.0.0.1:58290)
0928 14:43:26.245470 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.245524 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:26.245677 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:26.246427 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:26.246429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:26.246452 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:26.246527 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.246531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.247593 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.247597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.248822 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.248833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.249743 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.249747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.249922 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.249980 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 14:43:26.250097 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 14:43:26.250233 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":31,"thread_start_us":108,"threads_started":1}
I20260928 14:43:26.250385 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.244750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.1.201.3:44189)
0928 14:43:26.245289 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.245303 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.245603 (+ 300us) server_negotiation.cc:407] Connection header received
0928 14:43:26.245883 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.245891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.246095 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.246227 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:26.246637 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.247466 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.248930 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.249610 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.249736 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.249913 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.250140 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.250242 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":172,"threads_started":1}
I20260928 14:43:26.250679 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.245260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39559 (local address 127.0.0.1:57764)
0928 14:43:26.245905 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.245917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:26.245987 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:26.246706 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:26.246709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:26.246725 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:26.246800 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.246806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.248126 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.248129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.249295 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.249305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.250281 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.250285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.250379 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.250402 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:43:26.250446 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:43:26.250512 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260928 14:43:26.251190 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.245739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.1.201.2:39559)
0928 14:43:26.246240 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.246252 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.246273 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:43:26.246339 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.246343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.246543 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.246721 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:26.246939 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.247981 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.249422 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.250142 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.250783 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.250913 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.250965 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.251053 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":135,"threads_started":1}
I20260928 14:43:26.252485 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.244625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35723 (local address 127.0.0.1:38454)
0928 14:43:26.245469 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.245521 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:26.245665 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:26.246292 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:26.246299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:26.246323 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:26.246449 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.246455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.248028 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.248032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.250264 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.250273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.252112 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.252116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.252199 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.252220 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:43:26.252304 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:43:26.252381 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":997,"thread_start_us":56,"threads_started":1}
I20260928 14:43:26.252938 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.245351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.1.201.1:35723)
0928 14:43:26.245737 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.245755 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.245803 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:43:26.245883 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.245888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.246117 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.246600 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:26.246765 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.247878 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.250387 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.251183 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.251299 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.251448 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.252748 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.252836 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":89,"threads_started":1}
I20260928 14:43:26.253938 16012 tablet_service.cc:1467] Processing CreateTablet for tablet f09638cd69aa410e8a78a6595309edba (DEFAULT_TABLE table=test-table [id=cf3e9044cac04fe3a2d23c4a684c256b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:26.253939 15877 tablet_service.cc:1467] Processing CreateTablet for tablet f09638cd69aa410e8a78a6595309edba (DEFAULT_TABLE table=test-table [id=cf3e9044cac04fe3a2d23c4a684c256b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:26.255065 16012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09638cd69aa410e8a78a6595309edba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:26.255059 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09638cd69aa410e8a78a6595309edba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:26.257251 15742 tablet_service.cc:1467] Processing CreateTablet for tablet f09638cd69aa410e8a78a6595309edba (DEFAULT_TABLE table=test-table [id=cf3e9044cac04fe3a2d23c4a684c256b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:26.258296 15742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09638cd69aa410e8a78a6595309edba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:26.261518 16096 tablet_bootstrap.cc:492] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: Bootstrap starting.
I20260928 14:43:26.262161 16097 tablet_bootstrap.cc:492] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: Bootstrap starting.
I20260928 14:43:26.264744 16098 tablet_bootstrap.cc:492] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Bootstrap starting.
I20260928 14:43:26.264832 16097 tablet_bootstrap.cc:654] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:26.264832 16096 tablet_bootstrap.cc:654] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:26.265702 16097 log.cc:824] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:26.265702 16096 log.cc:824] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:26.266808 16098 tablet_bootstrap.cc:654] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:26.267439 16097 tablet_bootstrap.cc:492] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: No bootstrap required, opened a new log
I20260928 14:43:26.267638 16097 ts_tablet_manager.cc:1397] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 14:43:26.267688 16096 tablet_bootstrap.cc:492] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: No bootstrap required, opened a new log
I20260928 14:43:26.267858 16096 ts_tablet_manager.cc:1397] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260928 14:43:26.267956 16098 log.cc:824] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:26.270278 16098 tablet_bootstrap.cc:492] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: No bootstrap required, opened a new log
I20260928 14:43:26.270457 16098 ts_tablet_manager.cc:1397] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260928 14:43:26.271262 16097 raft_consensus.cc:348] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.271554 16097 raft_consensus.cc:374] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:26.271652 16097 raft_consensus.cc:729] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a99558758e243a58f97846f55e226ea, State: Initialized, Role: FOLLOWER
I20260928 14:43:26.271605 16096 raft_consensus.cc:348] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.271850 16096 raft_consensus.cc:374] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:26.271898 16096 raft_consensus.cc:729] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccee2c5a37d94de3b94683c4ac1ac6f0, State: Initialized, Role: FOLLOWER
I20260928 14:43:26.272156 16097 consensus_queue.cc:260] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.272289 16096 consensus_queue.cc:260] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.272997 16079 heartbeater.cc:503] Master 127.1.201.62:42193 was elected leader, sending a full tablet report...
I20260928 14:43:26.273250 16096 ts_tablet_manager.cc:1428] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260928 14:43:26.274225 16097 ts_tablet_manager.cc:1428] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260928 14:43:26.275827 15944 heartbeater.cc:503] Master 127.1.201.62:42193 was elected leader, sending a full tablet report...
I20260928 14:43:26.276540 16098 raft_consensus.cc:348] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.276880 16098 raft_consensus.cc:374] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:26.276994 16098 raft_consensus.cc:729] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af2e4d277844e1aaf9e918e942e3b68, State: Initialized, Role: FOLLOWER
I20260928 14:43:26.277400 16098 consensus_queue.cc:260] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.278080 15808 heartbeater.cc:503] Master 127.1.201.62:42193 was elected leader, sending a full tablet report...
I20260928 14:43:26.278323 16098 ts_tablet_manager.cc:1428] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260928 14:43:26.342013 16104 raft_consensus.cc:482] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:43:26.342263 16104 raft_consensus.cc:504] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.343714 16104 leader_election.cc:288] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189), 0a99558758e243a58f97846f55e226ea (127.1.201.2:39559)
I20260928 14:43:26.348464 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.344027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39559 (local address 127.1.201.1:51725)
0928 14:43:26.344450 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.344468 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:26.344543 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:26.344834 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:26.344838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:26.344870 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:26.344957 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.344961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.345806 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.345809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.347304 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.347315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.348126 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.348128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.348210 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.348231 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:43:26.348289 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:43:26.348344 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":140,"thread_start_us":105,"threads_started":1}
I20260928 14:43:26.348523 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.344159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:51725 (local address 127.1.201.2:39559)
0928 14:43:26.344301 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.344305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.344550 (+ 245us) server_negotiation.cc:407] Connection header received
0928 14:43:26.344660 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.344663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.344720 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.344792 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:26.345048 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.345693 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.347434 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.348002 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.348105 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.348166 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.348331 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.348398 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:43:26.349155 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.343867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:44189 (local address 127.1.201.1:38631)
0928 14:43:26.344208 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.344223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:26.344332 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:26.344622 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:26.344624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:26.344643 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:26.344753 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.344758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.345725 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.345727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.347304 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.347315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.348741 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.348746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.348860 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.348884 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:43:26.348959 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:43:26.349050 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":91,"thread_start_us":98,"threads_started":1}
I20260928 14:43:26.349234 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.344015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:38631 (local address 127.1.201.3:44189)
0928 14:43:26.344147 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.344152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.344310 (+ 158us) server_negotiation.cc:407] Connection header received
0928 14:43:26.344441 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.344444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.344488 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.344590 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:26.344899 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.345620 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.347885 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.348496 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.348600 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.348658 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.349012 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.349107 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:43:26.349318 15898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09638cd69aa410e8a78a6595309edba" candidate_uuid: "7af2e4d277844e1aaf9e918e942e3b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a99558758e243a58f97846f55e226ea" is_pre_election: true
I20260928 14:43:26.349453 16102 raft_consensus.cc:482] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:43:26.349620 16102 raft_consensus.cc:504] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.349773 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09638cd69aa410e8a78a6595309edba" candidate_uuid: "7af2e4d277844e1aaf9e918e942e3b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" is_pre_election: true
I20260928 14:43:26.350083 15898 raft_consensus.cc:2454] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af2e4d277844e1aaf9e918e942e3b68 in term 0.
I20260928 14:43:26.350116 16033 raft_consensus.cc:2454] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af2e4d277844e1aaf9e918e942e3b68 in term 0.
I20260928 14:43:26.350638 15694 leader_election.cc:302] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a99558758e243a58f97846f55e226ea, 7af2e4d277844e1aaf9e918e942e3b68, ccee2c5a37d94de3b94683c4ac1ac6f0; no voters:
I20260928 14:43:26.350901 16102 leader_election.cc:288] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189), 7af2e4d277844e1aaf9e918e942e3b68 (127.1.201.1:35723)
I20260928 14:43:26.350919 16104 raft_consensus.cc:2781] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:43:26.350993 16104 raft_consensus.cc:482] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:43:26.351035 16104 raft_consensus.cc:3037] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:26.352414 16104 raft_consensus.cc:504] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.352747 16104 leader_election.cc:288] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [CANDIDATE]: Term 1 election: Requested vote from peers ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189), 0a99558758e243a58f97846f55e226ea (127.1.201.2:39559)
I20260928 14:43:26.353044 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09638cd69aa410e8a78a6595309edba" candidate_uuid: "7af2e4d277844e1aaf9e918e942e3b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0"
I20260928 14:43:26.353116 15898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09638cd69aa410e8a78a6595309edba" candidate_uuid: "7af2e4d277844e1aaf9e918e942e3b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a99558758e243a58f97846f55e226ea"
I20260928 14:43:26.353156 16033 raft_consensus.cc:3037] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:26.353209 15898 raft_consensus.cc:3037] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:26.354458 15898 raft_consensus.cc:2454] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af2e4d277844e1aaf9e918e942e3b68 in term 1.
I20260928 14:43:26.354453 16033 raft_consensus.cc:2454] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af2e4d277844e1aaf9e918e942e3b68 in term 1.
I20260928 14:43:26.354805 15695 leader_election.cc:302] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a99558758e243a58f97846f55e226ea, 7af2e4d277844e1aaf9e918e942e3b68; no voters:
I20260928 14:43:26.355238 15946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.351022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:44189 (local address 127.1.201.2:50683)
0928 14:43:26.351248 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.351262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:26.351349 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:26.351711 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:26.351714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:26.351733 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:26.351825 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.351830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.352622 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.352625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.353732 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.353741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.354819 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.354826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.354926 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.354948 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:43:26.355053 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:43:26.355131 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260928 14:43:26.355458 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.351123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:50683 (local address 127.1.201.3:44189)
0928 14:43:26.351514 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.351517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.351529 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:43:26.351563 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.351565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.351607 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.351684 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:26.351947 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.352511 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.353905 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.354524 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.354714 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.354770 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.355084 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.355150 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260928 14:43:26.355850 16104 raft_consensus.cc:2781] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:43:26.355867 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09638cd69aa410e8a78a6595309edba" candidate_uuid: "0a99558758e243a58f97846f55e226ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" is_pre_election: true
I20260928 14:43:26.356040 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.351832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:37649 (local address 127.1.201.1:35723)
0928 14:43:26.351947 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.351951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.351961 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:43:26.351995 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.351998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.352034 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.352100 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:26.352397 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.352980 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.354638 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.355410 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.355520 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.355595 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.355878 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.355942 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:43:26.356051 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.351283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35723 (local address 127.1.201.2:37649)
0928 14:43:26.351582 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.351593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:26.351669 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:26.352137 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:26.352139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:26.352157 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:26.352275 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.352285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.353091 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.353094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.354253 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.354266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.355580 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.355587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.355707 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.355730 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:43:26.355860 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 14:43:26.355927 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":121,"threads_started":1}
I20260928 14:43:26.356057 16033 raft_consensus.cc:2379] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a99558758e243a58f97846f55e226ea in current term 1: Already voted for candidate 7af2e4d277844e1aaf9e918e942e3b68 in this term.
I20260928 14:43:26.356097 16104 raft_consensus.cc:686] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 LEADER]: Becoming Leader. State: Replica: 7af2e4d277844e1aaf9e918e942e3b68, State: Running, Role: LEADER
I20260928 14:43:26.356447 16104 consensus_queue.cc:237] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.356781 15762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09638cd69aa410e8a78a6595309edba" candidate_uuid: "0a99558758e243a58f97846f55e226ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af2e4d277844e1aaf9e918e942e3b68" is_pre_election: true
I20260928 14:43:26.358053 15830 leader_election.cc:302] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a99558758e243a58f97846f55e226ea; no voters: 7af2e4d277844e1aaf9e918e942e3b68, ccee2c5a37d94de3b94683c4ac1ac6f0
I20260928 14:43:26.358311 16102 raft_consensus.cc:2726] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 14:43:26.359506 15615 catalog_manager.cc:5495] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7af2e4d277844e1aaf9e918e942e3b68 (127.1.201.1). New cstate: current_term: 1 leader_uuid: "7af2e4d277844e1aaf9e918e942e3b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } health_report { overall_health: UNKNOWN } } }
I20260928 14:43:26.374234 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.370989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.1.201.1:35723)
0928 14:43:26.371151 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.371155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.371170 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:43:26.371247 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.371249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.371297 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.371373 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:43:26.371670 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.372171 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.372978 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.373177 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.373248 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.373474 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:43:26.373947 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:43:26.374035 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.374093 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.374134 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:43:26.379917 16033 raft_consensus.cc:1261] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 1 FOLLOWER]: Refusing update from remote peer 7af2e4d277844e1aaf9e918e942e3b68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:43:26.379953 15898 raft_consensus.cc:1261] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Refusing update from remote peer 7af2e4d277844e1aaf9e918e942e3b68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:43:26.380641 16108 consensus_queue.cc:1035] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:43:26.380878 16104 consensus_queue.cc:1035] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:43:26.387897 16113 mvcc.cc:205] Tried to move back new op lower bound from 7334324659725115392 to 7334324659638620160. Current Snapshot: MvccSnapshot[applied={T|T < 7334324659725115392}]
I20260928 14:43:26.389652 16112 mvcc.cc:205] Tried to move back new op lower bound from 7334324659725115392 to 7334324659638620160. Current Snapshot: MvccSnapshot[applied={T|T < 7334324659725115392}]
I20260928 14:43:26.395390 15616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:43:26.395520 15616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:43:26.398059 15616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Sending DeleteTablet for 3 replicas of tablet f09638cd69aa410e8a78a6595309edba
I20260928 14:43:26.398695 16012 tablet_service.cc:1514] Processing DeleteTablet for tablet f09638cd69aa410e8a78a6595309edba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:26 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:58290
I20260928 14:43:26.398834 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet f09638cd69aa410e8a78a6595309edba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:26 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:57764
I20260928 14:43:26.399199 16121 tablet_replica.cc:330] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: stopping tablet replica
I20260928 14:43:26.399142 15742 tablet_service.cc:1514] Processing DeleteTablet for tablet f09638cd69aa410e8a78a6595309edba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:26 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:38454
I20260928 14:43:26.399523 16121 raft_consensus.cc:2229] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:43:26.399639 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 15678
I20260928 14:43:26.400695 16120 tablet_replica.cc:330] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: stopping tablet replica
I20260928 14:43:26.400724 16121 raft_consensus.cc:2258] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:43:26.400992 16120 raft_consensus.cc:2229] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:43:26.401278 16120 raft_consensus.cc:2258] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:43:26.402168 16121 ts_tablet_manager.cc:1905] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:43:26.403152 16120 ts_tablet_manager.cc:1905] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:43:26.404793 16121 ts_tablet_manager.cc:1918] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:43:26.404894 16121 log.cc:1196] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/f09638cd69aa410e8a78a6595309edba
I20260928 14:43:26.405215 16121 ts_tablet_manager.cc:1939] T f09638cd69aa410e8a78a6595309edba P 0a99558758e243a58f97846f55e226ea: Deleting consensus metadata
I20260928 14:43:26.406332 15601 catalog_manager.cc:4841] TS 0a99558758e243a58f97846f55e226ea (127.1.201.2:39559): tablet f09638cd69aa410e8a78a6595309edba (table test-table [id=cf3e9044cac04fe3a2d23c4a684c256b]) successfully deleted
I20260928 14:43:26.406384 16120 ts_tablet_manager.cc:1918] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:43:26.406468 16120 log.cc:1196] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/f09638cd69aa410e8a78a6595309edba
I20260928 14:43:26.406795 16120 ts_tablet_manager.cc:1939] T f09638cd69aa410e8a78a6595309edba P ccee2c5a37d94de3b94683c4ac1ac6f0: Deleting consensus metadata
I20260928 14:43:26.407975 15600 catalog_manager.cc:4841] TS ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189): tablet f09638cd69aa410e8a78a6595309edba (table test-table [id=cf3e9044cac04fe3a2d23c4a684c256b]) successfully deleted
W20260928 14:43:26.408030 15601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:35723 (error 108)
I20260928 14:43:26.408156 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 15814
I20260928 14:43:26.413879 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 15949
I20260928 14:43:26.415004 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.414574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35723 (local address 127.0.0.1:38472)
0928 14:43:26.414817 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.414897 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260928 14:43:26.415509 15601 catalog_manager.cc:4584] TS 7af2e4d277844e1aaf9e918e942e3b68 (127.1.201.1:35723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f09638cd69aa410e8a78a6595309edba: Network error: Client connection negotiation failed: client connection to 127.1.201.1:35723: connect: Connection refused (error 111)
I20260928 14:43:26.419375 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 15584
I20260928 14:43:26.426285 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42193
--webserver_interface=127.1.201.62
--webserver_port=38177
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:42193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:26.534577 16124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:26.534858 16124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:26.534897 16124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:26.539037 16124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:43:26.539108 16124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:26.539127 16124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:26.539144 16124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:43:26.539160 16124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:43:26.544911 16124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:42193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=38177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:26.546056 16124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:26.547063 16124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:26.561086 16130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:26.561102 16133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:26.561156 16124 server_base.cc:1034] running on GCE node
W20260928 14:43:26.561084 16131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:26.561982 16124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:26.563149 16124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:26.564343 16124 hybrid_clock.cc:648] HybridClock initialized: now 1790606606564328 us; error 37 us; skew 500 ppm
Sep 28 14:43:26 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:26.567102 16124 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:43:26.568032 16124 webserver.cc:460] Webserver started at http://127.1.201.62:38177/ using document root <none> and password file <none>
I20260928 14:43:26.568611 16124 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:26.568729 16124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:26.572659 16124 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 14:43:26.574843 16140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:26.576072 16124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:43:26.576200 16124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "3303ce6b76224d5897a4710bb5dbc2ba"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "869fa5c74399849db664a5df9e384b0b"
server_key_iv: "d89b308a2eef9bd90cb8c7da6246ccc9"
server_key_version: "encryptionkey@0"
I20260928 14:43:26.576813 16124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:26.594889 16124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:26.597892 16124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:26.598083 16124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:26.605311 16124 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:42193
I20260928 14:43:26.605572 16192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:42193 every 8 connection(s)
I20260928 14:43:26.606470 16124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:43:26.610847 16193 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:43:26.612522 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 16124
I20260928 14:43:26.612525 16193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Bootstrap starting.
I20260928 14:43:26.612973 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:35723
--local_ip_for_outbound_sockets=127.1.201.1
--tserver_master_addrs=127.1.201.62:42193
--webserver_port=43823
--webserver_interface=127.1.201.1
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:43:26.629876 16193 log.cc:824] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:26.638703 16193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:43:26.639083 16193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Bootstrap complete.
I20260928 14:43:26.643177 16193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:26.643559 16193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3303ce6b76224d5897a4710bb5dbc2ba, State: Initialized, Role: FOLLOWER
I20260928 14:43:26.644045 16193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:26.644163 16193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:43:26.644261 16193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:43:26.644379 16193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:43:26.646309 16193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:26.646644 16193 leader_election.cc:302] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3303ce6b76224d5897a4710bb5dbc2ba; no voters:
I20260928 14:43:26.646970 16193 leader_election.cc:288] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:43:26.647053 16198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:43:26.647248 16198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 2 LEADER]: Becoming Leader. State: Replica: 3303ce6b76224d5897a4710bb5dbc2ba, State: Running, Role: LEADER
I20260928 14:43:26.647533 16198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:26.648181 16193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:43:26.648468 16199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3303ce6b76224d5897a4710bb5dbc2ba. Latest consensus state: current_term: 2 leader_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } } }
I20260928 14:43:26.648610 16199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:26.648963 16202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:43:26.649082 16200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } } }
I20260928 14:43:26.649258 16200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:26.653671 16202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:43:26.654321 16202 catalog_manager.cc:1259] Loaded cluster ID: 551865a3e4294f81aeed8a0f7f33d02d
I20260928 14:43:26.654383 16202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:43:26.655825 16202 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 14:43:26.660742 16202 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 14:43:26.666107 16202 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:43:26.666823 16202 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Loaded TSK: 0
I20260928 14:43:26.667965 16202 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:43:26.728085 16196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:26.728381 16196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:26.728458 16196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:26.732527 16196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:26.732610 16196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:26.732707 16196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:43:26.737985 16196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:35723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=43823
--tserver_master_addrs=127.1.201.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:26.739013 16196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:26.740005 16196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:26.750440 16222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:26.750476 16221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:26.750422 16224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:26.751024 16196 server_base.cc:1034] running on GCE node
I20260928 14:43:26.751479 16196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:26.752092 16196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:26.753283 16196 hybrid_clock.cc:648] HybridClock initialized: now 1790606606753257 us; error 41 us; skew 500 ppm
Sep 28 14:43:26 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:26.756346 16196 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:43:26.757263 16196 webserver.cc:460] Webserver started at http://127.1.201.1:43823/ using document root <none> and password file <none>
I20260928 14:43:26.757879 16196 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:26.757984 16196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:26.761960 16196 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:43:26.764245 16231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:26.765528 16196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:43:26.765664 16196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "7af2e4d277844e1aaf9e918e942e3b68"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "7b2fd83946c200d1115a298f7f51b5c9"
server_key_iv: "4a78b411c72998635fecc14c0a944c7a"
server_key_version: "encryptionkey@0"
I20260928 14:43:26.766264 16196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:26.788456 16196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:26.792747 16196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:26.792958 16196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:26.793709 16196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:26.795167 16238 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:43:26.800346 16196 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:43:26.800423 16196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260928 14:43:26.800495 16196 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:43:26.802966 16196 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:43:26.803030 16196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260928 14:43:26.803342 16238 tablet_bootstrap.cc:492] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Bootstrap starting.
I20260928 14:43:26.814730 16196 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:35723
I20260928 14:43:26.814877 16346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:35723 every 8 connection(s)
I20260928 14:43:26.815972 16196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:43:26.820014 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 16196
I20260928 14:43:26.820524 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:39559
--local_ip_for_outbound_sockets=127.1.201.2
--tserver_master_addrs=127.1.201.62:42193
--webserver_port=33687
--webserver_interface=127.1.201.2
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:43:26.823580 16238 log.cc:824] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:43:26 dist-test-slave-0n27 krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:26.832861 16238 tablet_bootstrap.cc:492] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:43:26.833294 16238 tablet_bootstrap.cc:492] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Bootstrap complete.
I20260928 14:43:26.834798 16238 ts_tablet_manager.cc:1397] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.008s
I20260928 14:43:26.838101 16238 raft_consensus.cc:348] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.838636 16238 raft_consensus.cc:729] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af2e4d277844e1aaf9e918e942e3b68, State: Initialized, Role: FOLLOWER
I20260928 14:43:26.839141 16238 consensus_queue.cc:260] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } }
I20260928 14:43:26.839886 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.818478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:36521 (local address 127.1.201.62:42193)
0928 14:43:26.818937 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.818958 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.820939 (+ 1981us) server_negotiation.cc:407] Connection header received
0928 14:43:26.821739 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.821756 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.822232 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.822730 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:26.824833 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.826483 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.828443 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.829235 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.829437 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.832803 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:26.832813 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:26.832830 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:26.832869 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:26.835391 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:26.836021 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:26.836027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:26.836032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:26.836167 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:26.836543 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:26.836546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:26.836547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:26.836969 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:26.837075 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.837972 (+ 897us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.838272 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":177,"threads_started":1}
I20260928 14:43:26.840085 16238 ts_tablet_manager.cc:1428] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:43:26.840859 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.818440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:36521)
0928 14:43:26.819746 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.819806 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:26.821125 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:26.823275 (+ 2150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:26.823284 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:26.823957 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:26.824635 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.824656 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.826679 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.826685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.828281 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.828295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.829661 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.829666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.829928 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.830634 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:26.830673 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:26.832566 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:26.835524 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:26.835530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:26.835543 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:26.835891 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:26.836282 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:26.836285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:26.836287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:26.836442 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:26.837135 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:26.837141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:26.837413 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 14:43:26.837609 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 14:43:26.838890 (+ 1281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":210,"threads_started":1}
I20260928 14:43:26.843407 16347 heartbeater.cc:347] Connected to a master server at 127.1.201.62:42193
I20260928 14:43:26.843631 16347 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:26.844146 16347 heartbeater.cc:511] Master 127.1.201.62:42193 requested a full tablet report, sending...
I20260928 14:43:26.851845 16157 ts_manager.cc:194] Registered new tserver with Master: 7af2e4d277844e1aaf9e918e942e3b68 (127.1.201.1:35723)
I20260928 14:43:26.856117 16157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:36521
I20260928 14:43:26.858913 16347 heartbeater.cc:503] Master 127.1.201.62:42193 was elected leader, sending a full tablet report...
I20260928 14:43:26.867188 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.856087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.1.201.1:35723)
0928 14:43:26.861997 (+ 5910us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:26.862009 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:26.862034 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:43:26.862097 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:26.862102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:26.862288 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:26.862432 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:26.862799 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.863866 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.865630 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.866434 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.866548 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.866678 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:26.866934 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 14:43:26.867056 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5814,"thread_start_us":91,"threads_started":1}
I20260928 14:43:26.867451 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:26.855282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35723 (local address 127.0.0.1:38482)
0928 14:43:26.856388 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:26.856433 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:26.856561 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:26.862474 (+ 5913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:26.862480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:26.862510 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:26.862663 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.862671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.863988 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.863993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.865479 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:26.865493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:26.866569 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:26.866574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:26.866719 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:26.866769 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 14:43:26.866888 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 14:43:26.867060 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":124,"threads_started":1}
I20260928 14:43:26.868570 16280 tablet_service.cc:1514] Processing DeleteTablet for tablet f09638cd69aa410e8a78a6595309edba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:26 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:38482
I20260928 14:43:26.869588 16356 tablet_replica.cc:330] stopping tablet replica
I20260928 14:43:26.869920 16356 raft_consensus.cc:2229] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:43:26.870173 16356 raft_consensus.cc:2258] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:43:26.872223 16356 ts_tablet_manager.cc:1905] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:43:26.876710 16356 ts_tablet_manager.cc:1918] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:43:26.876806 16356 log.cc:1196] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/f09638cd69aa410e8a78a6595309edba
I20260928 14:43:26.877084 16356 ts_tablet_manager.cc:1939] T f09638cd69aa410e8a78a6595309edba P 7af2e4d277844e1aaf9e918e942e3b68: Deleting consensus metadata
I20260928 14:43:26.879048 16142 catalog_manager.cc:4841] TS 7af2e4d277844e1aaf9e918e942e3b68 (127.1.201.1:35723): tablet f09638cd69aa410e8a78a6595309edba (table test-table [id=cf3e9044cac04fe3a2d23c4a684c256b]) successfully deleted
W20260928 14:43:26.947896 16351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:26.948204 16351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:26.948288 16351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:26.952572 16351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:26.952625 16351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:26.952725 16351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:43:26.958102 16351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:39559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=33687
--tserver_master_addrs=127.1.201.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:26.959137 16351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:26.960192 16351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:26.969921 16364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:26.969928 16361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:26.969990 16351 server_base.cc:1034] running on GCE node
W20260928 14:43:26.969930 16362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:26.970589 16351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:26.971199 16351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:26.972400 16351 hybrid_clock.cc:648] HybridClock initialized: now 1790606606972385 us; error 26 us; skew 500 ppm
Sep 28 14:43:26 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:26.975020 16351 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:43:26.975956 16351 webserver.cc:460] Webserver started at http://127.1.201.2:33687/ using document root <none> and password file <none>
I20260928 14:43:26.976505 16351 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:26.976611 16351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:26.980472 16351 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:43:26.982542 16371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:26.983668 16351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:43:26.983809 16351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "0a99558758e243a58f97846f55e226ea"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "5ec637d7547dac9eda14c00f23ac6ca7"
server_key_iv: "ea016a3e8dc49c60e4f1cae0973e311d"
server_key_version: "encryptionkey@0"
I20260928 14:43:26.984331 16351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:26.995164 16351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:26.997817 16351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:26.998026 16351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:26.998600 16351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:26.999374 16351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:26.999447 16351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:26.999512 16351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:26.999560 16351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:27.008349 16351 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:39559
I20260928 14:43:27.008379 16484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:39559 every 8 connection(s)
I20260928 14:43:27.009281 16351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:43:27.018543 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 16351
Sep 28 14:43:27 dist-test-slave-0n27 krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:27.019012 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:44189
--local_ip_for_outbound_sockets=127.1.201.3
--tserver_master_addrs=127.1.201.62:42193
--webserver_port=32921
--webserver_interface=127.1.201.3
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:43:27.023448 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.011034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:34321 (local address 127.1.201.62:42193)
0928 14:43:27.011173 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:27.011176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:27.011979 (+ 803us) server_negotiation.cc:407] Connection header received
0928 14:43:27.013194 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:27.013197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:27.013258 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:27.013364 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:27.014523 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.015078 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.016251 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.016497 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.016616 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.019398 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:27.019405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:27.019411 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:27.019448 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:27.021046 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:27.021705 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:27.021708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:27.021709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:27.021842 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:27.022271 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:27.022273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:27.022274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:27.022500 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:27.022575 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:27.023178 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 14:43:27.023327 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:43:27.024802 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.011415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:34321)
0928 14:43:27.011836 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.011875 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:27.012986 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:27.013544 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:27.013550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:27.013904 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:27.014372 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.014385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.015209 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.015213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.016120 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.016130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.016652 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.016657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.016829 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.017460 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:27.017475 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:27.019223 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:27.021203 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:27.021211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:27.021227 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:27.021594 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:27.021944 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:27.021949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:27.021952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:27.022140 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:27.022587 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:27.022596 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:27.022913 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 14:43:27.023122 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 14:43:27.023497 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
I20260928 14:43:27.026132 16485 heartbeater.cc:347] Connected to a master server at 127.1.201.62:42193
I20260928 14:43:27.026383 16485 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:27.026866 16485 heartbeater.cc:511] Master 127.1.201.62:42193 requested a full tablet report, sending...
I20260928 14:43:27.027935 16157 ts_manager.cc:194] Registered new tserver with Master: 0a99558758e243a58f97846f55e226ea (127.1.201.2:39559)
I20260928 14:43:27.028484 16157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:34321
W20260928 14:43:27.130079 16488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:27.130383 16488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:27.130460 16488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:27.134508 16488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:27.134589 16488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:27.134675 16488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:43:27.139981 16488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:44189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=32921
--tserver_master_addrs=127.1.201.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:27.141033 16488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:27.142140 16488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:27.152242 16496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:27.152248 16494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:27.152280 16488 server_base.cc:1034] running on GCE node
W20260928 14:43:27.152217 16493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:27.152832 16488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:27.153465 16488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:27.154651 16488 hybrid_clock.cc:648] HybridClock initialized: now 1790606607154633 us; error 28 us; skew 500 ppm
Sep 28 14:43:27 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:27.157461 16488 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:43:27.158411 16488 webserver.cc:460] Webserver started at http://127.1.201.3:32921/ using document root <none> and password file <none>
I20260928 14:43:27.158980 16488 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:27.159099 16488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:27.162851 16488 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 14:43:27.164964 16503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:27.166347 16488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:43:27.166467 16488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0"
format_stamp: "Formatted at 2026-09-28 14:43:26 on dist-test-slave-0n27"
server_key: "69a02bea76c5fdd01eb4cfa209f492f0"
server_key_iv: "f0b2a7e0d5398379f243fed83aaf2cd8"
server_key_version: "encryptionkey@0"
I20260928 14:43:27.167018 16488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:27.189412 16488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:27.192831 16488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:27.193042 16488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:27.193636 16488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:27.194489 16488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:27.194562 16488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:27.194629 16488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:27.194677 16488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:27.203626 16488 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:44189
I20260928 14:43:27.203651 16616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:44189 every 8 connection(s)
I20260928 14:43:27.204627 16488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:43:27.206789 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 16488
Sep 28 14:43:27 dist-test-slave-0n27 krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:27.216547 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.208015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.1.201.62:42193)
0928 14:43:27.208219 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:27.208224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:27.208453 (+ 229us) server_negotiation.cc:407] Connection header received
0928 14:43:27.208633 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:27.208636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:27.208697 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:27.208799 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:27.209558 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.210181 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.211057 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.211288 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.211397 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.212964 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:27.212968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:27.212975 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:27.212996 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:27.214618 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:27.215058 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:27.215063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:27.215065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:27.215150 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:27.215494 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:27.215499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:27.215504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:27.215766 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:27.215963 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:27.216194 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 14:43:27.216338 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260928 14:43:27.218948 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.206381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:39473 (local address 127.1.201.62:42193)
0928 14:43:27.206525 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:27.206529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:27.207447 (+ 918us) server_negotiation.cc:407] Connection header received
0928 14:43:27.208955 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:27.208958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:27.209007 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:27.209085 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:27.210281 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.210776 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.211983 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.212326 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.212448 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.215093 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:27.215097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:27.215099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:27.215138 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:27.216776 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:27.217375 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:27.217378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:27.217380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:27.217470 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:27.217873 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:27.217876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:27.217878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:27.218078 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:27.218176 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:27.218757 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 14:43:27.218850 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:43:27.219753 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.206794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:39473)
0928 14:43:27.207285 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.207337 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:27.208723 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:27.209261 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:27.209268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:27.209607 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:27.210122 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.210136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.210905 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.210908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.211850 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.211863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.212474 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.212477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.212627 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.213146 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:27.213162 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:27.214929 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:27.216889 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:27.216895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:27.216909 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:27.217271 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:27.217567 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:27.217571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:27.217572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:27.217735 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:27.218188 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:27.218193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:27.218507 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 14:43:27.218674 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 14:43:27.218911 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":127,"threads_started":1}
I20260928 14:43:27.220772 16617 heartbeater.cc:347] Connected to a master server at 127.1.201.62:42193
I20260928 14:43:27.220968 16617 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:27.221379 16617 heartbeater.cc:511] Master 127.1.201.62:42193 requested a full tablet report, sending...
I20260928 14:43:27.222313 16157 ts_manager.cc:194] Registered new tserver with Master: ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189)
I20260928 14:43:27.222899 16157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:39473
I20260928 14:43:27.229990 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:43:27.233918 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.230607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.1.201.62:42193)
0928 14:43:27.230721 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:27.230724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:27.230896 (+ 172us) server_negotiation.cc:407] Connection header received
0928 14:43:27.231015 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:27.231017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:27.231101 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:27.231197 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:43:27.231363 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.231831 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.232620 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.232825 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.232899 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.233214 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:43:27.233554 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:43:27.233650 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:27.233707 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 14:43:27.233748 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:43:27.234477 16157 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:43:27.237562 16157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:43:27.244972 16280 tablet_service.cc:1467] Processing CreateTablet for tablet b21fe1ee15324968bf8e7704b0f93629 (DEFAULT_TABLE table=test-table [id=863f84fe4d8e4a6586777534e1d85302]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:27.245554 16280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21fe1ee15324968bf8e7704b0f93629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:27.248539 16238 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Bootstrap starting.
I20260928 14:43:27.249073 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.243840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:44189 (local address 127.0.0.1:58304)
0928 14:43:27.244200 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.244218 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:27.244294 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:27.244958 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:27.244961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:27.244978 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:27.245073 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.245080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.246379 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.246383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.247481 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.247492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.248701 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.248705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.248818 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.248854 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:43:27.248917 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:43:27.248976 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":1}
I20260928 14:43:27.249493 16238 tablet_bootstrap.cc:654] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:27.249820 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.244109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39559 (local address 127.0.0.1:57770)
0928 14:43:27.244476 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.244488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:27.244563 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:27.245871 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:27.245875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:27.245893 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:27.245980 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.245985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.247037 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.247044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.248107 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.248121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.249344 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.249352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.249449 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.249470 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:43:27.249601 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 14:43:27.249674 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":93,"threads_started":1}
I20260928 14:43:27.249845 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.244015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.1.201.3:44189)
0928 14:43:27.244516 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:27.244529 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:27.244552 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:43:27.244631 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:27.244639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:27.244822 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:27.244986 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:27.245205 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.246240 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.247626 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.248490 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.249391 (+ 901us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.249541 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:27.249598 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 14:43:27.249683 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":175,"threads_started":1}
I20260928 14:43:27.250501 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.244309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.1.201.2:39559)
0928 14:43:27.245262 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:27.245273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:27.245294 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:43:27.245365 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:27.245370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:27.245548 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:27.245688 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:27.246104 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.246909 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.248254 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.249021 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.249130 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.249353 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:27.249607 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 14:43:27.249702 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":120,"threads_started":1}
I20260928 14:43:27.250715 16238 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: No bootstrap required, opened a new log
I20260928 14:43:27.250814 16238 ts_tablet_manager.cc:1397] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:43:27.251390 16238 raft_consensus.cc:348] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.251531 16238 raft_consensus.cc:374] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:27.251569 16238 raft_consensus.cc:729] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af2e4d277844e1aaf9e918e942e3b68, State: Initialized, Role: FOLLOWER
I20260928 14:43:27.251688 16238 consensus_queue.cc:260] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.251938 16238 ts_tablet_manager.cc:1428] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:43:27.253052 16551 tablet_service.cc:1467] Processing CreateTablet for tablet b21fe1ee15324968bf8e7704b0f93629 (DEFAULT_TABLE table=test-table [id=863f84fe4d8e4a6586777534e1d85302]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:27.253185 16419 tablet_service.cc:1467] Processing CreateTablet for tablet b21fe1ee15324968bf8e7704b0f93629 (DEFAULT_TABLE table=test-table [id=863f84fe4d8e4a6586777534e1d85302]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:27.254187 16551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21fe1ee15324968bf8e7704b0f93629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:27.254261 16419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21fe1ee15324968bf8e7704b0f93629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:27.260584 16630 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Bootstrap starting.
I20260928 14:43:27.260649 16631 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Bootstrap starting.
I20260928 14:43:27.263074 16631 tablet_bootstrap.cc:654] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:27.263150 16630 tablet_bootstrap.cc:654] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:27.263787 16631 log.cc:824] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:27.263895 16630 log.cc:824] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:27.265478 16630 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: No bootstrap required, opened a new log
I20260928 14:43:27.265489 16631 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: No bootstrap required, opened a new log
I20260928 14:43:27.265686 16630 ts_tablet_manager.cc:1397] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:43:27.265692 16631 ts_tablet_manager.cc:1397] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:43:27.268730 16630 raft_consensus.cc:348] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.268754 16631 raft_consensus.cc:348] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.268966 16630 raft_consensus.cc:374] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:27.268960 16631 raft_consensus.cc:374] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:27.269023 16631 raft_consensus.cc:729] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccee2c5a37d94de3b94683c4ac1ac6f0, State: Initialized, Role: FOLLOWER
I20260928 14:43:27.269032 16630 raft_consensus.cc:729] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a99558758e243a58f97846f55e226ea, State: Initialized, Role: FOLLOWER
I20260928 14:43:27.269433 16630 consensus_queue.cc:260] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.269433 16631 consensus_queue.cc:260] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.270076 16485 heartbeater.cc:503] Master 127.1.201.62:42193 was elected leader, sending a full tablet report...
I20260928 14:43:27.270109 16617 heartbeater.cc:503] Master 127.1.201.62:42193 was elected leader, sending a full tablet report...
I20260928 14:43:27.270406 16631 ts_tablet_manager.cc:1428] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:43:27.270406 16630 ts_tablet_manager.cc:1428] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 14:43:27.317317 16348 tablet.cc:2369] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:43:27.404699 16634 raft_consensus.cc:482] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:43:27.404892 16634 raft_consensus.cc:504] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.405958 16634 leader_election.cc:288] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a99558758e243a58f97846f55e226ea (127.1.201.2:39559), 7af2e4d277844e1aaf9e918e942e3b68 (127.1.201.1:35723)
I20260928 14:43:27.409819 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.406221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39559 (local address 127.1.201.3:45069)
0928 14:43:27.406336 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.406354 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:27.406456 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:27.406791 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:27.406796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:27.406814 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:27.406957 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.406964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.407875 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.407878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.408681 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.408690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.409345 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.409351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.409454 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.409482 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 14:43:27.409551 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:43:27.409640 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 14:43:27.409839 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.406319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:45069 (local address 127.1.201.2:39559)
0928 14:43:27.406488 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:27.406493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:27.406509 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:43:27.406564 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:27.406566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:27.406633 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:27.407062 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:27.407195 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.407763 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.408812 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.409223 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.409300 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.409381 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:27.409595 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 14:43:27.409671 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:43:27.409855 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.406517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35723 (local address 127.1.201.3:57975)
0928 14:43:27.406834 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.406847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:27.406918 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:27.407152 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:27.407155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:27.407171 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:27.407276 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.407281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.408006 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.408008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.408726 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:27.408759 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.409413 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.409417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:27.409504 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.409521 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 14:43:27.409616 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:43:27.409676 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":136,"threads_started":1}
I20260928 14:43:27.409894 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.406662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:57975 (local address 127.1.201.1:35723)
0928 14:43:27.406893 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:27.406897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:27.406909 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:43:27.407014 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:27.407016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:27.407053 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:27.407121 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:27.407383 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.407901 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.408876 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.409289 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.409391 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.409453 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:27.409652 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 14:43:27.409728 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":58,"threads_started":1}
I20260928 14:43:27.410468 16439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21fe1ee15324968bf8e7704b0f93629" candidate_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a99558758e243a58f97846f55e226ea" is_pre_election: true
I20260928 14:43:27.410455 16300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21fe1ee15324968bf8e7704b0f93629" candidate_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af2e4d277844e1aaf9e918e942e3b68" is_pre_election: true
I20260928 14:43:27.410776 16439 raft_consensus.cc:2454] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccee2c5a37d94de3b94683c4ac1ac6f0 in term 0.
I20260928 14:43:27.410776 16300 raft_consensus.cc:2454] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccee2c5a37d94de3b94683c4ac1ac6f0 in term 0.
I20260928 14:43:27.411250 16505 leader_election.cc:302] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a99558758e243a58f97846f55e226ea, ccee2c5a37d94de3b94683c4ac1ac6f0; no voters:
I20260928 14:43:27.411495 16634 raft_consensus.cc:2781] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:43:27.411573 16634 raft_consensus.cc:482] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:43:27.411604 16634 raft_consensus.cc:3037] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:27.412688 16634 raft_consensus.cc:504] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.413041 16634 leader_election.cc:288] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [CANDIDATE]: Term 1 election: Requested vote from peers 0a99558758e243a58f97846f55e226ea (127.1.201.2:39559), 7af2e4d277844e1aaf9e918e942e3b68 (127.1.201.1:35723)
I20260928 14:43:27.413437 16439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21fe1ee15324968bf8e7704b0f93629" candidate_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a99558758e243a58f97846f55e226ea"
I20260928 14:43:27.413437 16300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21fe1ee15324968bf8e7704b0f93629" candidate_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af2e4d277844e1aaf9e918e942e3b68"
I20260928 14:43:27.413560 16439 raft_consensus.cc:3037] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:27.413560 16300 raft_consensus.cc:3037] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:27.414863 16300 raft_consensus.cc:2454] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccee2c5a37d94de3b94683c4ac1ac6f0 in term 1.
I20260928 14:43:27.414863 16439 raft_consensus.cc:2454] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccee2c5a37d94de3b94683c4ac1ac6f0 in term 1.
I20260928 14:43:27.415146 16505 leader_election.cc:302] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7af2e4d277844e1aaf9e918e942e3b68, ccee2c5a37d94de3b94683c4ac1ac6f0; no voters:
I20260928 14:43:27.415330 16634 raft_consensus.cc:2781] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:43:27.415544 16634 raft_consensus.cc:686] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 1 LEADER]: Becoming Leader. State: Replica: ccee2c5a37d94de3b94683c4ac1ac6f0, State: Running, Role: LEADER
I20260928 14:43:27.415922 16634 consensus_queue.cc:237] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.419049 16157 catalog_manager.cc:5495] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3). New cstate: current_term: 1 leader_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } health_report { overall_health: UNKNOWN } } }
I20260928 14:43:27.444883 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.441837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.1.201.3:44189)
0928 14:43:27.441943 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:27.441946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:27.441957 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:43:27.442028 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:27.442030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:27.442081 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:27.442167 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:43:27.442373 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.442854 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.443614 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:27.443823 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:27.443894 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:27.444156 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:43:27.444558 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:43:27.444647 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:27.444709 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 14:43:27.444759 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:43:27.450882 16439 raft_consensus.cc:1261] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Refusing update from remote peer ccee2c5a37d94de3b94683c4ac1ac6f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:43:27.450876 16300 raft_consensus.cc:1261] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Refusing update from remote peer ccee2c5a37d94de3b94683c4ac1ac6f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:43:27.451531 16634 consensus_queue.cc:1035] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:43:27.451756 16638 consensus_queue.cc:1035] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:43:27.455489 16618 tablet.cc:2369] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:43:27.465983 16156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:43:27.466171 16156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:43:27.468538 16156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Sending DeleteTablet for 3 replicas of tablet b21fe1ee15324968bf8e7704b0f93629
I20260928 14:43:27.469321 16280 tablet_service.cc:1514] Processing DeleteTablet for tablet b21fe1ee15324968bf8e7704b0f93629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:27 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:38482
I20260928 14:43:27.469398 16551 tablet_service.cc:1514] Processing DeleteTablet for tablet b21fe1ee15324968bf8e7704b0f93629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:27 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:58304
I20260928 14:43:27.469525 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 16196
I20260928 14:43:27.469817 16419 tablet_service.cc:1514] Processing DeleteTablet for tablet b21fe1ee15324968bf8e7704b0f93629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:27 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:57770
I20260928 14:43:27.470500 16651 tablet_replica.cc:330] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: stopping tablet replica
I20260928 14:43:27.470830 16651 raft_consensus.cc:2229] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:43:27.471120 16651 raft_consensus.cc:2258] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:43:27.471572 16652 tablet_replica.cc:330] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: stopping tablet replica
I20260928 14:43:27.471856 16652 raft_consensus.cc:2229] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:43:27.472203 16652 raft_consensus.cc:2258] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:43:27.473747 16652 ts_tablet_manager.cc:1905] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:43:27.476395 16142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:35723 (error 108)
I20260928 14:43:27.476693 16652 ts_tablet_manager.cc:1918] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:43:27.476797 16652 log.cc:1196] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/b21fe1ee15324968bf8e7704b0f93629
I20260928 14:43:27.477140 16652 ts_tablet_manager.cc:1939] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Deleting consensus metadata
W20260928 14:43:27.477681 16627 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:43:27.477374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35723 (local address 127.0.0.1:36634)
0928 14:43:27.477501 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.477576 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:35723: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:43:27.477913 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 16351
I20260928 14:43:27.478775 16141 catalog_manager.cc:4841] TS ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189): tablet b21fe1ee15324968bf8e7704b0f93629 (table test-table [id=863f84fe4d8e4a6586777534e1d85302]) successfully deleted
W20260928 14:43:27.480891 16142 catalog_manager.cc:4584] TS 7af2e4d277844e1aaf9e918e942e3b68 (127.1.201.1:35723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b21fe1ee15324968bf8e7704b0f93629: Network error: Client connection negotiation failed: client connection to 127.1.201.1:35723: connect: Connection reset by peer (error 104)
I20260928 14:43:27.486001 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 16488
I20260928 14:43:27.486955 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.485851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39559 (local address 127.0.0.1:47472)
0928 14:43:27.486795 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.486861 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.2:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":848}
I20260928 14:43:27.491644 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 16124
I20260928 14:43:27.498191 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42193
--webserver_interface=127.1.201.62
--webserver_port=38177
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:42193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:27.610457 16654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:27.610760 16654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:27.610831 16654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:27.614832 16654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:43:27.614913 16654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:27.614952 16654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:27.615020 16654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:43:27.615058 16654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:43:27.620359 16654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:42193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=38177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:27.621388 16654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:27.622421 16654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:27.631016 16660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:27.631083 16654 server_base.cc:1034] running on GCE node
W20260928 14:43:27.631060 16662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:27.631100 16659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:27.631834 16654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:27.632931 16654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:27.634123 16654 hybrid_clock.cc:648] HybridClock initialized: now 1790606607634079 us; error 56 us; skew 500 ppm
Sep 28 14:43:27 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:27.636911 16654 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:43:27.637869 16654 webserver.cc:460] Webserver started at http://127.1.201.62:38177/ using document root <none> and password file <none>
I20260928 14:43:27.638419 16654 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:27.638505 16654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:27.642182 16654 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 14:43:27.644308 16669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:27.645552 16654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:43:27.645702 16654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "3303ce6b76224d5897a4710bb5dbc2ba"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "869fa5c74399849db664a5df9e384b0b"
server_key_iv: "d89b308a2eef9bd90cb8c7da6246ccc9"
server_key_version: "encryptionkey@0"
I20260928 14:43:27.646301 16654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:27.661866 16654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:27.665001 16654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:27.665211 16654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:27.672227 16654 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:42193
I20260928 14:43:27.672515 16721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:42193 every 8 connection(s)
I20260928 14:43:27.673300 16654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:43:27.674044 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 16654
I20260928 14:43:27.674530 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:35723
--local_ip_for_outbound_sockets=127.1.201.1
--tserver_master_addrs=127.1.201.62:42193
--webserver_port=43823
--webserver_interface=127.1.201.1
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:43:27.678642 16722 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:43:27.680169 16722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Bootstrap starting.
I20260928 14:43:27.697098 16722 log.cc:824] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:27.708137 16722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:43:27.708531 16722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba: Bootstrap complete.
I20260928 14:43:27.712556 16722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:27.712934 16722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3303ce6b76224d5897a4710bb5dbc2ba, State: Initialized, Role: FOLLOWER
I20260928 14:43:27.713410 16722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:27.713521 16722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:43:27.713614 16722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:43:27.713729 16722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:43:27.715610 16722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:27.715945 16722 leader_election.cc:302] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3303ce6b76224d5897a4710bb5dbc2ba; no voters:
I20260928 14:43:27.716279 16722 leader_election.cc:288] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:43:27.716360 16726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:43:27.716544 16726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [term 3 LEADER]: Becoming Leader. State: Replica: 3303ce6b76224d5897a4710bb5dbc2ba, State: Running, Role: LEADER
I20260928 14:43:27.716832 16726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } }
I20260928 14:43:27.717482 16722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:43:27.717711 16727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } } }
I20260928 14:43:27.717887 16727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:27.718228 16728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3303ce6b76224d5897a4710bb5dbc2ba. Latest consensus state: current_term: 3 leader_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303ce6b76224d5897a4710bb5dbc2ba" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42193 } } }
I20260928 14:43:27.718994 16728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3303ce6b76224d5897a4710bb5dbc2ba [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:27.718989 16730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:43:27.724120 16730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:43:27.724778 16730 catalog_manager.cc:1259] Loaded cluster ID: 551865a3e4294f81aeed8a0f7f33d02d
I20260928 14:43:27.724836 16730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 14:43:27.725826 16730 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efcf46649ad google::LogMessage::Fail() at ??:0
@ 0x7efcf466788a google::LogMessage::SendToLog() at ??:0
@ 0x7efcf46644de google::LogMessage::Flush() at ??:0
@ 0x7efcf4665a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efcfefa7fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7efcfefa8d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efcfefa5579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7efcfefe9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7efcf68b63d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efcf68b6c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7efcf68b83f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7efcf68a7512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7efcf7eeb6db start_thread at ??:0
@ 0x7efcf316261f clone at ??:0
W20260928 14:43:27.813722 16724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:27.814041 16724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:27.814117 16724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:27.818166 16724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:27.818216 16724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:27.818306 16724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:43:27.823817 16724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:35723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=43823
--tserver_master_addrs=127.1.201.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:27.824846 16724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:27.825978 16724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:27.835596 16764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:27.835596 16766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:27.835561 16763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:27.836771 16724 server_base.cc:1034] running on GCE node
I20260928 14:43:27.837199 16724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:27.837858 16724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:27.839047 16724 hybrid_clock.cc:648] HybridClock initialized: now 1790606607839018 us; error 49 us; skew 500 ppm
Sep 28 14:43:27 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:27.841961 16724 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:43:27.842873 16724 webserver.cc:460] Webserver started at http://127.1.201.1:43823/ using document root <none> and password file <none>
I20260928 14:43:27.843437 16724 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:27.843497 16724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:27.847443 16724 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 14:43:27.849874 16773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:27.851193 16724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:43:27.851333 16724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "7af2e4d277844e1aaf9e918e942e3b68"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "7b2fd83946c200d1115a298f7f51b5c9"
server_key_iv: "4a78b411c72998635fecc14c0a944c7a"
server_key_version: "encryptionkey@0"
I20260928 14:43:27.851890 16724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:27.870891 16724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:27.874157 16724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:27.874368 16724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:27.874963 16724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:27.876123 16780 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:43:27.881546 16724 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:43:27.881642 16724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 14:43:27.881724 16724 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:43:27.883401 16724 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:43:27.883472 16724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 14:43:27.883778 16780 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Bootstrap starting.
I20260928 14:43:27.894615 16724 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:35723
I20260928 14:43:27.894771 16889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:35723 every 8 connection(s)
I20260928 14:43:27.895870 16724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:43:27.902097 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 16724
I20260928 14:43:27.902590 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:39559
--local_ip_for_outbound_sockets=127.1.201.2
--tserver_master_addrs=127.1.201.62:42193
--webserver_port=33687
--webserver_interface=127.1.201.2
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:43:27.905476 16780 log.cc:824] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:27.912231 16780 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:43:27.912603 16780 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Bootstrap complete.
I20260928 14:43:27.913719 16780 ts_tablet_manager.cc:1397] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.013s
I20260928 14:43:27.916023 16780 raft_consensus.cc:348] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.916435 16780 raft_consensus.cc:729] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af2e4d277844e1aaf9e918e942e3b68, State: Initialized, Role: FOLLOWER
I20260928 14:43:27.916955 16780 consensus_queue.cc:260] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:27.917970 16780 ts_tablet_manager.cc:1428] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260928 14:43:27.935137 16892 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:43:27.898268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:58547)
0928 14:43:27.898845 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.898903 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:27.899942 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:27.934215 (+ 34273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":145,"threads_started":1}
W20260928 14:43:27.935745 16890 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:42193 (0 consecutive failures): Network error: Failed to ping master at 127.1.201.62:42193: Client connection negotiation failed: client connection to 127.1.201.62:42193: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:43:27.936457 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.936186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:42925)
0928 14:43:27.936328 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.936398 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 14:43:27.937081 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:27.936865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:51981)
0928 14:43:27.936978 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:27.937029 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 14:43:27.937281 16890 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:43:28.022943 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:28.023181 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:28.023272 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:28.027271 16893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:28.027328 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:28.027437 16893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:43:28.032794 16893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:39559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=33687
--tserver_master_addrs=127.1.201.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:28.033834 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:28.034833 16893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:28.044399 16901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:28.044448 16893 server_base.cc:1034] running on GCE node
W20260928 14:43:28.044395 16900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:28.044391 16903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:28.045075 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:28.045768 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:28.047000 16893 hybrid_clock.cc:648] HybridClock initialized: now 1790606608046980 us; error 34 us; skew 500 ppm
Sep 28 14:43:28 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606608, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:28.049722 16893 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:43:28.050552 16893 webserver.cc:460] Webserver started at http://127.1.201.2:33687/ using document root <none> and password file <none>
I20260928 14:43:28.051061 16893 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:28.051117 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:28.054773 16893 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 14:43:28.057029 16910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:28.058321 16893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260928 14:43:28.058442 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "0a99558758e243a58f97846f55e226ea"
format_stamp: "Formatted at 2026-09-28 14:43:25 on dist-test-slave-0n27"
server_key: "5ec637d7547dac9eda14c00f23ac6ca7"
server_key_iv: "ea016a3e8dc49c60e4f1cae0973e311d"
server_key_version: "encryptionkey@0"
I20260928 14:43:28.058974 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:28.068969 16893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:28.072108 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:28.072319 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:28.072904 16893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:28.073992 16917 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:43:28.077905 16893 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:43:28.077975 16893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 14:43:28.078038 16893 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:43:28.079650 16893 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:43:28.079736 16893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 14:43:28.079968 16917 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Bootstrap starting.
I20260928 14:43:28.090273 16893 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:39559
I20260928 14:43:28.090368 17025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:39559 every 8 connection(s)
I20260928 14:43:28.091228 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:43:28.094475 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:28.093098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:49899)
0928 14:43:28.093583 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:28.093706 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":146,"threads_started":1}
W20260928 14:43:28.094913 17026 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:42193 (0 consecutive failures): Network error: Failed to ping master at 127.1.201.62:42193: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
I20260928 14:43:28.095517 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:28.095315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:33037)
0928 14:43:28.095427 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:28.095469 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:43:28.096269 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:28.096077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:37963)
0928 14:43:28.096179 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:28.096222 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:43:28.096481 17026 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:43:28.097585 16917 log.cc:824] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:28.100211 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 16893
I20260928 14:43:28.100677 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:44189
--local_ip_for_outbound_sockets=127.1.201.3
--tserver_master_addrs=127.1.201.62:42193
--webserver_port=32921
--webserver_interface=127.1.201.3
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:43:28.104908 16917 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:43:28.105273 16917 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Bootstrap complete.
I20260928 14:43:28.106482 16917 ts_tablet_manager.cc:1397] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.007s
I20260928 14:43:28.108981 16917 raft_consensus.cc:348] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:28.109385 16917 raft_consensus.cc:729] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a99558758e243a58f97846f55e226ea, State: Initialized, Role: FOLLOWER
I20260928 14:43:28.109970 16917 consensus_queue.cc:260] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:28.110978 16917 ts_tablet_manager.cc:1428] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 14:43:28.111184 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:28.110884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:58689)
0928 14:43:28.111037 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:28.111103 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 14:43:28.111830 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:28.111607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:47587)
0928 14:43:28.111728 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:28.111774 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 14:43:28.210804 17030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:28.211062 17030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:28.211107 17030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:28.215052 17030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:28.215101 17030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:28.215154 17030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:43:28.220176 17030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:44189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=32921
--tserver_master_addrs=127.1.201.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:28.221148 17030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:28.222205 17030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:43:28.231859 17030 server_base.cc:1034] running on GCE node
W20260928 14:43:28.231881 17039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:28.231838 17037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:28.231881 17036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:28.232501 17030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:28.233096 17030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:28.234289 17030 hybrid_clock.cc:648] HybridClock initialized: now 1790606608234275 us; error 25 us; skew 500 ppm
Sep 28 14:43:28 dist-test-slave-0n27 krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606608, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:28.236879 17030 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:43:28.237715 17030 webserver.cc:460] Webserver started at http://127.1.201.3:32921/ using document root <none> and password file <none>
I20260928 14:43:28.238294 17030 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:28.238387 17030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:28.242116 17030 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 14:43:28.244110 17046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:28.245370 17030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:43:28.245510 17030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0"
format_stamp: "Formatted at 2026-09-28 14:43:26 on dist-test-slave-0n27"
server_key: "69a02bea76c5fdd01eb4cfa209f492f0"
server_key_iv: "f0b2a7e0d5398379f243fed83aaf2cd8"
server_key_version: "encryptionkey@0"
I20260928 14:43:28.246093 17030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:28.258117 17030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:28.260891 17030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:28.261083 17030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:28.261685 17030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:28.262547 17030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:28.262636 17030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:28.262702 17030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:28.262737 17030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:28.271596 17030 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:44189
I20260928 14:43:28.271616 17160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:44189 every 8 connection(s)
I20260928 14:43:28.272555 17030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:43:28.276260 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:28.274570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:48767)
0928 14:43:28.275001 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:28.275482 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
W20260928 14:43:28.276733 17161 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:42193 (0 consecutive failures): Network error: Failed to ping master at 127.1.201.62:42193: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
I20260928 14:43:28.277201 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 17030
I20260928 14:43:28.277395 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:28.277139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:47981)
0928 14:43:28.277274 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:28.277334 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:43:28.278100 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:28.277888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:51983)
0928 14:43:28.277986 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:28.278031 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:43:28.278287 17161 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:43:28.938335 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:28.937938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:45027)
0928 14:43:28.938174 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:28.938228 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260928 14:43:29.113024 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:29.112560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:48943)
0928 14:43:29.112858 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:29.112916 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
I20260928 14:43:29.279114 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:29.278760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:53257)
0928 14:43:29.278972 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:29.279021 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260928 14:43:29.319762 17168 raft_consensus.cc:482] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:43:29.319980 17168 raft_consensus.cc:504] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:29.321125 17168 leader_election.cc:288] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189), 0a99558758e243a58f97846f55e226ea (127.1.201.2:39559)
Sep 28 14:43:29 dist-test-slave-0n27 krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.3@KRBTEST.COM
Sep 28 14:43:29 dist-test-slave-0n27 krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.2@KRBTEST.COM
I20260928 14:43:29.336174 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:29.321695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:42889 (local address 127.1.201.2:39559)
0928 14:43:29.322033 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:29.322043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:29.322072 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:43:29.322283 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:29.322289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:29.322742 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:29.323110 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:29.324748 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:29.326307 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:29.328218 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:29.328958 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:29.329132 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:29.331991 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:29.331996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:29.332007 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:29.332033 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:29.333900 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:29.334366 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:29.334372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:29.334377 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:29.334505 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:29.334787 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:29.334789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:29.334791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:29.335219 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:29.335339 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:29.335744 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 14:43:29.335992 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":127,"threads_started":1}
I20260928 14:43:29.336158 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:29.321325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:44189 (local address 127.1.201.1:52665)
0928 14:43:29.321435 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:29.321452 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:29.321557 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:29.323576 (+ 2019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:29.323579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:29.323772 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:29.324542 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:29.324563 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:29.326467 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:29.326471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:29.328092 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:29.328103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:29.329101 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:29.329108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:29.329320 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:29.329996 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:29.330012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:29.331523 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:29.333601 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:29.333609 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:29.333625 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:29.333985 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:29.334354 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:29.334357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:29.334359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:29.334513 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:29.335158 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:29.335164 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:29.335417 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 14:43:29.335631 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 14:43:29.335978 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":296,"spinlock_wait_cycles":50816}
I20260928 14:43:29.336351 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:29.321469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:52665 (local address 127.1.201.3:44189)
0928 14:43:29.321858 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:29.321886 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:29.321932 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:43:29.322879 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:29.322887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:29.323307 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:29.323607 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:29.324748 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:29.326307 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:29.328218 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:29.328958 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:29.329132 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:29.331799 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:29.331806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:29.331822 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:29.331851 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:29.333480 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:29.334092 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:29.334100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:29.334106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:29.334245 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:29.334607 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:29.334610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:29.334611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:29.335046 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:29.335144 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:29.335765 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 14:43:29.336035 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
I20260928 14:43:29.336663 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:29.321337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:39559 (local address 127.1.201.1:42889)
0928 14:43:29.321755 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:29.321771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:29.321904 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:29.323181 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:29.323186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:29.323748 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:29.324542 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:29.324563 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:29.326467 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:29.326471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:29.328092 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:29.328103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:29.329328 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:29.329331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:29.329437 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:29.330165 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:29.330177 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:29.331847 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:29.334000 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:29.334003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:29.334004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:29.334264 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:29.334607 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:29.334610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:29.334612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:29.334698 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:29.335322 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:29.335324 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:29.335417 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 14:43:29.335612 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 14:43:29.336061 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":427,"thread_start_us":90,"threads_started":1}
I20260928 14:43:29.338333 16980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21fe1ee15324968bf8e7704b0f93629" candidate_uuid: "7af2e4d277844e1aaf9e918e942e3b68" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a99558758e243a58f97846f55e226ea" is_pre_election: true
I20260928 14:43:29.338696 16980 raft_consensus.cc:2454] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af2e4d277844e1aaf9e918e942e3b68 in term 1.
W20260928 14:43:29.339042 16774 leader_election.cc:341] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189): Not found: Tablet not found: b21fe1ee15324968bf8e7704b0f93629
I20260928 14:43:29.339529 16775 leader_election.cc:302] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a99558758e243a58f97846f55e226ea, 7af2e4d277844e1aaf9e918e942e3b68; no voters: ccee2c5a37d94de3b94683c4ac1ac6f0
I20260928 14:43:29.339839 17168 raft_consensus.cc:2781] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:43:29.339946 17168 raft_consensus.cc:482] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:43:29.340011 17168 raft_consensus.cc:3037] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:43:29.342244 17168 raft_consensus.cc:504] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:29.342547 17168 leader_election.cc:288] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [CANDIDATE]: Term 2 election: Requested vote from peers ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189), 0a99558758e243a58f97846f55e226ea (127.1.201.2:39559)
I20260928 14:43:29.343142 16980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21fe1ee15324968bf8e7704b0f93629" candidate_uuid: "7af2e4d277844e1aaf9e918e942e3b68" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a99558758e243a58f97846f55e226ea"
I20260928 14:43:29.343261 16980 raft_consensus.cc:3037] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:43:29.345209 16980 raft_consensus.cc:2454] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af2e4d277844e1aaf9e918e942e3b68 in term 2.
I20260928 14:43:29.345559 16775 leader_election.cc:302] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a99558758e243a58f97846f55e226ea, 7af2e4d277844e1aaf9e918e942e3b68; no voters:
W20260928 14:43:29.345654 16774 leader_election.cc:341] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189): Not found: Tablet not found: b21fe1ee15324968bf8e7704b0f93629
I20260928 14:43:29.345764 17168 raft_consensus.cc:2781] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:43:29.345989 17168 raft_consensus.cc:686] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [term 2 LEADER]: Becoming Leader. State: Replica: 7af2e4d277844e1aaf9e918e942e3b68, State: Running, Role: LEADER
I20260928 14:43:29.346325 17168 consensus_queue.cc:237] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:29.346674 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:29.346427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:46713)
0928 14:43:29.346544 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:29.346594 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 14:43:29.855048 16980 raft_consensus.cc:1261] T b21fe1ee15324968bf8e7704b0f93629 P 0a99558758e243a58f97846f55e226ea [term 2 FOLLOWER]: Refusing update from remote peer 7af2e4d277844e1aaf9e918e942e3b68: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 14:43:29.855875 17174 consensus_queue.cc:1035] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 14:43:29.858073 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:29.857675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:34209)
0928 14:43:29.857945 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:29.857995 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":48,"threads_started":1}
I20260928 14:43:29.860730 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:29.860261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:45491)
0928 14:43:29.860553 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:29.860631 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":63,"threads_started":1}
I20260928 14:43:29.860816 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:29.860513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:32877)
0928 14:43:29.860679 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:29.860735 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 14:43:29.860810 16774 consensus_peers.cc:487] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68 -> Peer ccee2c5a37d94de3b94683c4ac1ac6f0 (127.1.201.3:44189): Couldn't send request to peer ccee2c5a37d94de3b94683c4ac1ac6f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b21fe1ee15324968bf8e7704b0f93629. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:43:30.280238 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:30.279839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:36097)
0928 14:43:30.280089 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:30.280150 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260928 14:43:30.394145 17183 ts_tablet_manager.cc:927] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Initiating tablet copy from peer 7af2e4d277844e1aaf9e918e942e3b68 (127.1.201.1:35723)
I20260928 14:43:30.394793 17183 tablet_copy_client.cc:316] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: tablet copy: Beginning tablet copy session from remote peer at address 127.1.201.1:35723
Sep 28 14:43:30 dist-test-slave-0n27 krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606608, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:43:30.411456 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:30.395307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35723 (local address 127.1.201.3:45461)
0928 14:43:30.395465 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:30.395485 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:30.395677 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:30.396231 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:30.396237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:30.396649 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:30.396760 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:30.396766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:30.397740 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:30.397745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:30.398436 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:30.398445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:30.399033 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:30.399036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:30.399155 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:30.406248 (+ 7093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:30.406261 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:30.407739 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:30.409529 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:30.409538 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:30.409545 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:30.409869 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:30.410195 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:30.410199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:30.410201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:30.410288 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:30.410775 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:30.410782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:30.410915 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 14:43:30.411059 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 14:43:30.411241 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 14:43:30.411466 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:30.395505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:45461 (local address 127.1.201.1:35723)
0928 14:43:30.395818 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:30.395825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:30.395845 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:43:30.395888 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:30.395893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:30.396078 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:30.396236 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:30.396889 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:30.397622 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:30.398561 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:30.398908 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:30.399028 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:30.407894 (+ 8866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:30.407903 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:30.407916 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:30.407941 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:30.409391 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:30.409984 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:30.409995 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:30.409996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:30.410077 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:30.410389 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:30.410392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:30.410394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:30.410659 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:30.410804 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:30.411165 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 14:43:30.411316 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":67,"threads_started":1}
I20260928 14:43:30.412722 16863 tablet_copy_service.cc:140] P 7af2e4d277844e1aaf9e918e942e3b68: Received BeginTabletCopySession request for tablet b21fe1ee15324968bf8e7704b0f93629 from peer ccee2c5a37d94de3b94683c4ac1ac6f0 ({username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:45461)
I20260928 14:43:30.412918 16863 tablet_copy_service.cc:161] P 7af2e4d277844e1aaf9e918e942e3b68: Beginning new tablet copy session on tablet b21fe1ee15324968bf8e7704b0f93629 from peer ccee2c5a37d94de3b94683c4ac1ac6f0 at {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:45461: session id = ccee2c5a37d94de3b94683c4ac1ac6f0-b21fe1ee15324968bf8e7704b0f93629
I20260928 14:43:30.414238 16863 tablet_copy_source_session.cc:205] T b21fe1ee15324968bf8e7704b0f93629 P 7af2e4d277844e1aaf9e918e942e3b68: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:43:30.418192 17183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21fe1ee15324968bf8e7704b0f93629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:30.424602 17183 tablet_copy_client.cc:791] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: tablet copy: Starting download of 0 data blocks...
I20260928 14:43:30.424805 17183 tablet_copy_client.cc:655] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: tablet copy: Starting download of 1 WAL segments...
I20260928 14:43:30.426513 17183 tablet_copy_client.cc:531] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:43:30.428037 17183 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Bootstrap starting.
I20260928 14:43:30.432996 17183 log.cc:824] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:30.439353 17183 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:43:30.439633 17183 tablet_bootstrap.cc:492] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Bootstrap complete.
I20260928 14:43:30.439922 17183 ts_tablet_manager.cc:1397] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260928 14:43:30.442116 17183 raft_consensus.cc:348] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:30.442368 17183 raft_consensus.cc:729] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccee2c5a37d94de3b94683c4ac1ac6f0, State: Initialized, Role: FOLLOWER
I20260928 14:43:30.442862 17183 consensus_queue.cc:260] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccee2c5a37d94de3b94683c4ac1ac6f0" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 44189 } } peers { permanent_uuid: "0a99558758e243a58f97846f55e226ea" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 39559 } } peers { permanent_uuid: "7af2e4d277844e1aaf9e918e942e3b68" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35723 } }
I20260928 14:43:30.443735 17183 ts_tablet_manager.cc:1428] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 14:43:30.444125 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:30.443877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:58781)
0928 14:43:30.444003 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:30.444064 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 14:43:30.444451 16863 tablet_copy_service.cc:342] P 7af2e4d277844e1aaf9e918e942e3b68: Request end of tablet copy session ccee2c5a37d94de3b94683c4ac1ac6f0-b21fe1ee15324968bf8e7704b0f93629 received from {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:45461
I20260928 14:43:30.444579 16863 tablet_copy_service.cc:433] P 7af2e4d277844e1aaf9e918e942e3b68: ending tablet copy session ccee2c5a37d94de3b94683c4ac1ac6f0-b21fe1ee15324968bf8e7704b0f93629 on tablet b21fe1ee15324968bf8e7704b0f93629 with peer ccee2c5a37d94de3b94683c4ac1ac6f0
I20260928 14:43:30.444829 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:30.444609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:60597)
0928 14:43:30.444736 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:30.444784 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:43:30.861929 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:30.861455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:46989)
0928 14:43:30.861699 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:30.861765 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":63,"threads_started":1}
I20260928 14:43:30.861967 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:30.861491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:35559)
0928 14:43:30.861752 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:30.861870 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":91,"threads_started":1}
I20260928 14:43:30.941560 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:30.941296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:34679)
0928 14:43:30.941425 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:30.941484 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:43:30.941715 17115 raft_consensus.cc:1203] T b21fe1ee15324968bf8e7704b0f93629 P ccee2c5a37d94de3b94683c4ac1ac6f0 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 14:43:30.942713 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:30.942471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:36313)
0928 14:43:30.942584 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:30.942638 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:43:31.863068 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:31.862598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:59003)
0928 14:43:31.862921 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:31.862976 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":77,"threads_started":1}
I20260928 14:43:31.942538 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:31.942169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:44679)
0928 14:43:31.942396 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:31.942447 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260928 14:43:31.943689 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:31.943290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:37507)
0928 14:43:31.943542 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:31.943588 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260928 14:43:32.864127 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:32.863743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:60923)
0928 14:43:32.863975 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:32.864038 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260928 14:43:32.943831 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:32.943450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:56083)
0928 14:43:32.943691 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:32.943742 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260928 14:43:32.944527 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:32.944335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:35277)
0928 14:43:32.944436 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:32.944478 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 14:43:32.944736 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:32.944340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:60057)
0928 14:43:32.944549 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:32.944636 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20260928 14:43:33.865434 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:33.865038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:50921)
0928 14:43:33.865272 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:33.865341 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260928 14:43:33.945745 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:33.945266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:50951)
0928 14:43:33.945562 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:33.945630 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
I20260928 14:43:33.946112 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:33.945700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:39711)
0928 14:43:33.945959 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:33.946019 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
I20260928 14:43:34.866689 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:34.866280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:46573)
0928 14:43:34.866532 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:34.866593 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
I20260928 14:43:34.947039 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:34.946536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:55011)
0928 14:43:34.946875 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:34.946946 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":160,"threads_started":1}
I20260928 14:43:34.947222 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:34.946802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:37203)
0928 14:43:34.947056 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:34.947105 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260928 14:43:35.867884 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:35.867471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:39973)
0928 14:43:35.867732 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:35.867789 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260928 14:43:35.948366 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:35.947921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:45007)
0928 14:43:35.948200 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:35.948257 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260928 14:43:35.948402 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:35.947921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:39877)
0928 14:43:35.948256 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:35.948307 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":93,"threads_started":1}
I20260928 14:43:36.869195 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:36.868749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:34449)
0928 14:43:36.869036 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:36.869088 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":103,"threads_started":1}
I20260928 14:43:36.949571 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:36.949153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:35249)
0928 14:43:36.949414 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:36.949479 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":98,"threads_started":1}
I20260928 14:43:36.949640 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:36.949188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:49575)
0928 14:43:36.949427 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:36.949489 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":93,"threads_started":1}
I20260928 14:43:37.870455 17210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:37.870043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:33633)
0928 14:43:37.870304 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:37.870362 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260928 14:43:37.950793 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:37.950328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:38173)
0928 14:43:37.950627 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:37.950694 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":125,"threads_started":1}
I20260928 14:43:37.950961 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:37.950539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:42493)
0928 14:43:37.950784 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:37.950836 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260928 14:43:38.871790 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:38.871361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:53321)
0928 14:43:38.871635 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:38.871693 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260928 14:43:38.952188 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:38.951710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:42297)
0928 14:43:38.951984 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:38.952050 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
I20260928 14:43:38.952189 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:38.951715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:45289)
0928 14:43:38.952061 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:38.952110 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":95,"threads_started":1}
I20260928 14:43:39.873081 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:39.872535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:34785)
0928 14:43:39.872868 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:39.872946 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260928 14:43:39.953387 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:39.952986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:45011)
0928 14:43:39.953240 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:39.953293 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260928 14:43:39.953476 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:39.952981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:60957)
0928 14:43:39.953315 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:39.953366 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":90,"threads_started":1}
I20260928 14:43:40.874279 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:40.873887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:51381)
0928 14:43:40.874129 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:40.874185 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260928 14:43:40.954524 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:40.954125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:58775)
0928 14:43:40.954375 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:40.954427 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
I20260928 14:43:40.954605 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:40.954192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:53769)
0928 14:43:40.954459 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:40.954513 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":61,"threads_started":1}
I20260928 14:43:41.875334 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:41.874956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:52923)
0928 14:43:41.875191 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:41.875242 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260928 14:43:41.955736 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:41.955301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:59703)
0928 14:43:41.955581 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:41.955638 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260928 14:43:41.955739 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:41.955335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:34629)
0928 14:43:41.955599 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:41.955655 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260928 14:43:42.876452 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:42.876033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.2:54069)
0928 14:43:42.876285 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:42.876359 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260928 14:43:42.957000 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:42.956599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.3:56949)
0928 14:43:42.956854 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:42.956907 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
I20260928 14:43:42.957297 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:42.956781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42193 (local address 127.1.201.1:45363)
0928 14:43:42.957123 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:42.957188 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":195,"threads_started":1}
I20260928 14:43:43.280203 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 16724
I20260928 14:43:43.286091 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 16893
I20260928 14:43:43.291661 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 17030
2026-09-28T14:43:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19138 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:43:43 dist-test-slave-0n27 krb5kdc[17232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:43:43 dist-test-slave-0n27 krb5kdc[17232](info): set up 2 sockets
Sep 28 14:43:43 dist-test-slave-0n27 krb5kdc[17232](info): commencing operation
krb5kdc: starting...
W20260928 14:43:44.319063 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:43:44 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:43:44.437652 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:34987
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:34987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:44.543954 17246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:44.544262 17246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:44.544338 17246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:44.548471 17246 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:43:44.548552 17246 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:43:44.548578 17246 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:43:44.548619 17246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:43:44.548667 17246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:44.548709 17246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:44.548743 17246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:43:44.548777 17246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:43:44.553862 17246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:34987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:44.554899 17246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:44.555923 17246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:44.564486 17251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:44.564486 17252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:44.564642 17254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:44.564532 17246 server_base.cc:1034] running on GCE node
I20260928 14:43:44.565240 17246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:44.566342 17246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:44.567523 17246 hybrid_clock.cc:648] HybridClock initialized: now 1790606624567477 us; error 57 us; skew 500 ppm
Sep 28 14:43:44 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606624, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:44.570215 17246 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:43:44.571059 17246 webserver.cc:460] Webserver started at http://127.1.201.62:35771/ using document root <none> and password file <none>
I20260928 14:43:44.571579 17246 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:44.571699 17246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:44.571936 17246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:44.574009 17246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab"
format_stamp: "Formatted at 2026-09-28 14:43:44 on dist-test-slave-0n27"
server_key: "f823842a0f54d7efccec6e2385dea35a"
server_key_iv: "d0d4a4e7f74b50c73ec9b91239353c67"
server_key_version: "encryptionkey@0"
I20260928 14:43:44.574555 17246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab"
format_stamp: "Formatted at 2026-09-28 14:43:44 on dist-test-slave-0n27"
server_key: "f823842a0f54d7efccec6e2385dea35a"
server_key_iv: "d0d4a4e7f74b50c73ec9b91239353c67"
server_key_version: "encryptionkey@0"
I20260928 14:43:44.578148 17246 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:43:44.580374 17261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:44.581625 17246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:43:44.581753 17246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab"
format_stamp: "Formatted at 2026-09-28 14:43:44 on dist-test-slave-0n27"
server_key: "f823842a0f54d7efccec6e2385dea35a"
server_key_iv: "d0d4a4e7f74b50c73ec9b91239353c67"
server_key_version: "encryptionkey@0"
I20260928 14:43:44.581890 17246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:44.610074 17246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:44.612884 17246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:44.613107 17246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:44.620203 17246 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:34987
I20260928 14:43:44.620492 17313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:34987 every 8 connection(s)
I20260928 14:43:44.621150 17246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:43:44.623217 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 17246
I20260928 14:43:44.623347 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:43:44.623623 1828 external_mini_cluster.cc:1428] Setting key d209ae00257efdc5e6c64409aff48970
I20260928 14:43:44.624094 17314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:44.631067 17314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Bootstrap starting.
Sep 28 14:43:44 dist-test-slave-0n27 krb5kdc[17232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:44.633349 17314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:44.634312 17314 log.cc:824] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:44.636484 17314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: No bootstrap required, opened a new log
I20260928 14:43:44.638162 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:44.624994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.1.201.62:34987)
0928 14:43:44.625406 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:44.625426 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:44.625468 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:43:44.626260 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:44.626281 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:44.626696 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:44.627065 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:44.627678 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:44.629130 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:44.630037 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:44.630732 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:44.630877 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:44.633304 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:44.633312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:44.633328 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:44.633360 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:44.635194 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:44.635592 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:44.635598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:44.635603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:44.635766 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:44.636011 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:44.636014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:44.636016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:44.636485 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:44.636677 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:44.636967 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 14:43:44.637245 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
I20260928 14:43:44.640527 17314 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:43:44.640725 17314 raft_consensus.cc:374] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:44.640758 17314 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca3d307a25a4e14a42e6d6d57a7b1ab, State: Initialized, Role: FOLLOWER
I20260928 14:43:44.641132 17314 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:43:44.641248 17314 raft_consensus.cc:388] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:43:44.641292 17314 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:43:44.641359 17314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:44.642220 17314 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:43:44.642493 17314 leader_election.cc:302] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca3d307a25a4e14a42e6d6d57a7b1ab; no voters:
I20260928 14:43:44.642731 17314 leader_election.cc:288] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:43:44.642856 17319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:43:44.643086 17319 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 1 LEADER]: Becoming Leader. State: Replica: bca3d307a25a4e14a42e6d6d57a7b1ab, State: Running, Role: LEADER
I20260928 14:43:44.643477 17319 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:43:44.643950 17314 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:43:44.645004 17320 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } } }
I20260928 14:43:44.645120 17320 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:44.645367 17321 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca3d307a25a4e14a42e6d6d57a7b1ab. Latest consensus state: current_term: 1 leader_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } } }
I20260928 14:43:44.645442 17321 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: This master's current role is: LEADER
I20260928 14:43:44.646278 17329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:43:44.648947 17329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:43:44.654086 17329 catalog_manager.cc:1347] Generated new cluster ID: 82ec9cb8498146019336476102339e3e
I20260928 14:43:44.654143 17329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:43:44.684860 17329 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:43:44.686132 17329 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:43:44.693871 17329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Generated new TSK 0
I20260928 14:43:44.694566 17329 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:43:44.733135 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34987
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:44.865859 17340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:44.866113 17340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:44.866160 17340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:44.869899 17340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:44.869944 17340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:44.869997 17340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:43:44.874907 17340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:44.875810 17340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:44.876725 17340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:44.885407 17348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:44.885479 17345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:44.885407 17346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:44.885578 17340 server_base.cc:1034] running on GCE node
I20260928 14:43:44.886106 17340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:44.886628 17340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:44.887785 17340 hybrid_clock.cc:648] HybridClock initialized: now 1790606624887772 us; error 22 us; skew 500 ppm
Sep 28 14:43:44 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606624, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:44.890549 17340 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:43:44.891420 17340 webserver.cc:460] Webserver started at http://127.1.201.1:45681/ using document root <none> and password file <none>
I20260928 14:43:44.892022 17340 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:44.892120 17340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:44.892349 17340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:44.894500 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf"
format_stamp: "Formatted at 2026-09-28 14:43:44 on dist-test-slave-0n27"
server_key: "5203a68597435e3531724e9c733bcda7"
server_key_iv: "dfce6399cebc7ff523f5854686c00f8e"
server_key_version: "encryptionkey@0"
I20260928 14:43:44.895077 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf"
format_stamp: "Formatted at 2026-09-28 14:43:44 on dist-test-slave-0n27"
server_key: "5203a68597435e3531724e9c733bcda7"
server_key_iv: "dfce6399cebc7ff523f5854686c00f8e"
server_key_version: "encryptionkey@0"
I20260928 14:43:44.899046 17340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 14:43:44.901459 17355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:44.902858 17340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:43:44.903007 17340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf"
format_stamp: "Formatted at 2026-09-28 14:43:44 on dist-test-slave-0n27"
server_key: "5203a68597435e3531724e9c733bcda7"
server_key_iv: "dfce6399cebc7ff523f5854686c00f8e"
server_key_version: "encryptionkey@0"
I20260928 14:43:44.903142 17340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:44.924456 17340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:44.927488 17340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:44.927716 17340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:44.928325 17340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:44.929219 17340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:44.929293 17340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:44.929369 17340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:44.929425 17340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:44.938632 17340 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:42921
I20260928 14:43:44.938643 17469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:42921 every 8 connection(s)
I20260928 14:43:44.939563 17340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:43:44.939925 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 17340
I20260928 14:43:44.940076 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:43:44.940405 1828 external_mini_cluster.cc:1428] Setting key 78298cafbd69741f1b5864b65911e78d
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
Sep 28 14:43:44 dist-test-slave-0n27 krb5kdc[17232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606624, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
I20260928 14:43:44.960399 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:44.941463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:37307 (local address 127.1.201.62:34987)
0928 14:43:44.941587 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:44.941590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:44.942761 (+ 1171us) server_negotiation.cc:407] Connection header received
0928 14:43:44.944558 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:44.944562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:44.944617 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:44.944689 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:44.946237 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:44.947058 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:44.948855 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:44.949321 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:44.949438 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:44.956205 (+ 6767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:44.956209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:44.956212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:44.956243 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:44.957963 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:44.958609 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:44.958613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:44.958615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:44.958712 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:44.959027 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:44.959030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:44.959031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:44.959300 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:44.959423 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:44.960003 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 14:43:44.960169 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:43:44.961055 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:44.942040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:37307)
0928 14:43:44.942578 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:44.942636 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:44.944328 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:44.944868 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:44.944875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:44.945322 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:44.946074 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:44.946096 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:44.947212 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:44.947216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:44.948656 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:44.948666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:44.949663 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:44.949669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:44.949903 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:44.953739 (+ 3836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:44.953761 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:44.955966 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:44.958103 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:44.958109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:44.958143 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:44.958485 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:44.958803 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:44.958805 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:44.958806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:44.958935 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:44.959414 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:44.959419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:44.959646 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 14:43:44.959826 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 14:43:44.960090 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":135,"threads_started":1}
I20260928 14:43:44.962404 17470 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34987
I20260928 14:43:44.962661 17470 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:44.963186 17470 heartbeater.cc:511] Master 127.1.201.62:34987 requested a full tablet report, sending...
I20260928 14:43:44.964682 17278 ts_manager.cc:194] Registered new tserver with Master: 4cac3ddab4ff465bb0fe08a49e6bfcaf (127.1.201.1:42921)
I20260928 14:43:44.966198 17278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:37307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:43:44.974176 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34987
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:45.081887 17475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:45.082253 17475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:45.082314 17475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:45.086315 17475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:45.086378 17475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:45.086431 17475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:43:45.091271 17475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:45.092265 17475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:45.093201 17475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:45.102701 17483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:45.102697 17480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:45.102736 17481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:45.102880 17475 server_base.cc:1034] running on GCE node
I20260928 14:43:45.103405 17475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:45.104027 17475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:45.105221 17475 hybrid_clock.cc:648] HybridClock initialized: now 1790606625105206 us; error 31 us; skew 500 ppm
Sep 28 14:43:45 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:45.108161 17475 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:43:45.109016 17475 webserver.cc:460] Webserver started at http://127.1.201.2:33379/ using document root <none> and password file <none>
I20260928 14:43:45.109613 17475 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:45.109674 17475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:45.109925 17475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:45.111898 17475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "444a46ca7b14413da09a9027603aa7a2"
format_stamp: "Formatted at 2026-09-28 14:43:45 on dist-test-slave-0n27"
server_key: "7ae12581a3d0204e11d56108346bbf0f"
server_key_iv: "4fc5bb64675b24bca9db72d5f5f0984c"
server_key_version: "encryptionkey@0"
I20260928 14:43:45.112418 17475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "444a46ca7b14413da09a9027603aa7a2"
format_stamp: "Formatted at 2026-09-28 14:43:45 on dist-test-slave-0n27"
server_key: "7ae12581a3d0204e11d56108346bbf0f"
server_key_iv: "4fc5bb64675b24bca9db72d5f5f0984c"
server_key_version: "encryptionkey@0"
I20260928 14:43:45.116216 17475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:43:45.118440 17490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:45.119580 17475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:43:45.119726 17475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "444a46ca7b14413da09a9027603aa7a2"
format_stamp: "Formatted at 2026-09-28 14:43:45 on dist-test-slave-0n27"
server_key: "7ae12581a3d0204e11d56108346bbf0f"
server_key_iv: "4fc5bb64675b24bca9db72d5f5f0984c"
server_key_version: "encryptionkey@0"
I20260928 14:43:45.119827 17475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:45.138617 17475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:45.141748 17475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:45.142012 17475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:45.142597 17475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:45.143450 17475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:45.143496 17475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:45.143616 17475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:45.143658 17475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:45.152486 17475 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:35771
I20260928 14:43:45.152554 17604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:35771 every 8 connection(s)
I20260928 14:43:45.153371 17475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:43:45.159962 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 17475
I20260928 14:43:45.160071 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:43:45.160365 1828 external_mini_cluster.cc:1428] Setting key 50cb0fab89fa0a643bff4b221e419525
Sep 28 14:43:45 dist-test-slave-0n27 krb5kdc[17232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:45.166482 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.155154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:55521 (local address 127.1.201.62:34987)
0928 14:43:45.155314 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:45.155319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:45.156110 (+ 791us) server_negotiation.cc:407] Connection header received
0928 14:43:45.157270 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:45.157273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:45.157317 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:45.157385 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:45.158598 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.159132 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.160289 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.160569 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.160680 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.163156 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:45.163160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:45.163163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:45.163221 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:45.164563 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:45.165048 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:45.165052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:45.165054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:45.165128 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:45.165462 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:45.165466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:45.165468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:45.165624 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:45.165721 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:45.166217 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 14:43:45.166329 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:43:45.167273 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.155528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:55521)
0928 14:43:45.155964 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:45.156005 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:45.157076 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:45.157537 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:45.157543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:45.157946 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:45.158449 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.158463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.159253 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.159255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.160163 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.160171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.160724 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.160728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.160942 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.161532 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:45.161547 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:45.162997 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:45.164695 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:45.164699 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:45.164712 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:45.164942 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:45.165221 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:45.165225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:45.165227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:45.165376 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:45.165722 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:45.165729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:45.165980 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 14:43:45.166168 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 14:43:45.166450 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
I20260928 14:43:45.168349 17605 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34987
I20260928 14:43:45.168603 17605 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:45.169056 17605 heartbeater.cc:511] Master 127.1.201.62:34987 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
I20260928 14:43:45.170120 17278 ts_manager.cc:194] Registered new tserver with Master: 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771)
I20260928 14:43:45.170648 17278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:55521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:43:45.186208 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34987
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:43:45.296934 17610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:43:45.297159 17610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:43:45.297240 17610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:43:45.301466 17610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:43:45.301517 17610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:43:45.301613 17610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:43:45.306736 17610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:43:45.307729 17610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:43:45.308751 17610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:43:45.318040 17618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:45.318078 17610 server_base.cc:1034] running on GCE node
W20260928 14:43:45.318112 17615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:43:45.318046 17616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:43:45.318619 17610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:43:45.319242 17610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:43:45.320418 17610 hybrid_clock.cc:648] HybridClock initialized: now 1790606625320403 us; error 32 us; skew 500 ppm
Sep 28 14:43:45 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:43:45.323146 17610 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:43:45.324077 17610 webserver.cc:460] Webserver started at http://127.1.201.3:41459/ using document root <none> and password file <none>
I20260928 14:43:45.324620 17610 fs_manager.cc:359] Metadata directory not provided
I20260928 14:43:45.324707 17610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:43:45.324918 17610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:43:45.326885 17610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "4908a7b69cd2411ba7e69b6785aeddb7"
format_stamp: "Formatted at 2026-09-28 14:43:45 on dist-test-slave-0n27"
server_key: "76d1449e594f6b0dee29a967c5477e00"
server_key_iv: "ef3821166a470226f96c840fb7ae02d8"
server_key_version: "encryptionkey@0"
I20260928 14:43:45.327442 17610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "4908a7b69cd2411ba7e69b6785aeddb7"
format_stamp: "Formatted at 2026-09-28 14:43:45 on dist-test-slave-0n27"
server_key: "76d1449e594f6b0dee29a967c5477e00"
server_key_iv: "ef3821166a470226f96c840fb7ae02d8"
server_key_version: "encryptionkey@0"
I20260928 14:43:45.331003 17610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:43:45.333364 17625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:45.334638 17610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:43:45.334774 17610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "4908a7b69cd2411ba7e69b6785aeddb7"
format_stamp: "Formatted at 2026-09-28 14:43:45 on dist-test-slave-0n27"
server_key: "76d1449e594f6b0dee29a967c5477e00"
server_key_iv: "ef3821166a470226f96c840fb7ae02d8"
server_key_version: "encryptionkey@0"
I20260928 14:43:45.334894 17610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:43:45.359550 17610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:43:45.362483 17610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:43:45.362677 17610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:43:45.363284 17610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:43:45.364046 17610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:43:45.364133 17610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:45.364217 17610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:43:45.364269 17610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:43:45.372967 17610 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:41945
I20260928 14:43:45.373003 17739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:41945 every 8 connection(s)
I20260928 14:43:45.373908 17610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:43:45.382064 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 17610
I20260928 14:43:45.382200 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:43:45.382543 1828 external_mini_cluster.cc:1428] Setting key 5cfb6eb473654127c403834def6d542a
Sep 28 14:43:45 dist-test-slave-0n27 krb5kdc[17232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:43:45.387372 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.375646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:55263 (local address 127.1.201.62:34987)
0928 14:43:45.375794 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:45.375797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:45.376523 (+ 726us) server_negotiation.cc:407] Connection header received
0928 14:43:45.377802 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:45.377805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:45.377863 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:45.377945 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:45.379058 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.379585 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.380667 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.380876 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.380981 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.383743 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:45.383747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:45.383750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:45.383782 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:45.385264 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:45.385797 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:45.385803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:45.385805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:45.385889 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:45.386284 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:45.386289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:45.386291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:45.386503 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:45.386623 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:45.387090 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 14:43:45.387219 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:43:45.388110 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.375972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:55263)
0928 14:43:45.376389 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:45.376428 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:45.377585 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:45.378113 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:45.378119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:45.378459 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:43:45.378917 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.378930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.379718 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.379733 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.380549 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.380557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.381014 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.381018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.381203 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.381840 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:43:45.381863 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:43:45.383567 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:43:45.385379 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:45.385384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:45.385398 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:45.385651 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:45.385985 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:43:45.385989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:43:45.385991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:43:45.386180 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:43:45.386623 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:43:45.386631 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:43:45.386864 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 14:43:45.387054 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 14:43:45.387375 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":129,"threads_started":1}
I20260928 14:43:45.389142 17740 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34987
I20260928 14:43:45.389400 17740 heartbeater.cc:464] Registering TS with master...
I20260928 14:43:45.389907 17740 heartbeater.cc:511] Master 127.1.201.62:34987 requested a full tablet report, sending...
I20260928 14:43:45.390833 17278 ts_manager.cc:194] Registered new tserver with Master: 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945)
I20260928 14:43:45.391355 17278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:55263
I20260928 14:43:45.396170 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:43:45.405329 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.398201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.1.201.62:34987)
0928 14:43:45.398317 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:45.398321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:45.398364 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:43:45.398464 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:45.398466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:45.398500 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:45.398567 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:43:45.399259 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.399745 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.400578 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.400863 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.401013 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.402418 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:43:45.402421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:43:45.402424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:43:45.402446 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:43:45.403892 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:45.404327 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:45.404330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:45.404331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:45.404409 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:43:45.404666 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:43:45.404668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:43:45.404669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:43:45.404859 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:43:45.404954 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:45.405123 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:43:45.405214 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:43:45.408416 17278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:43:45.412786 17278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:43:45.430802 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.423976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:35771 (local address 127.0.0.1:34966)
0928 14:43:45.424518 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:45.424590 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:45.424720 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:45.425873 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:45.425879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:45.425904 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:45.426060 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.426068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.427675 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.427678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.428963 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.428977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.430177 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.430184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.430323 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.430386 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 14:43:45.430484 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 14:43:45.430624 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260928 14:43:45.431269 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.425309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42921 (local address 127.0.0.1:34294)
0928 14:43:45.425881 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:45.425895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:45.425982 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:45.426905 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:45.426908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:45.426926 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:45.427016 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.427023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.428331 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.428334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.429575 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.429585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.430814 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.430817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.430949 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.430970 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:43:45.431040 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:43:45.431108 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":43,"thread_start_us":82,"threads_started":1}
I20260928 14:43:45.431605 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.424914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.1.201.2:35771)
0928 14:43:45.425265 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:45.425278 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:45.425302 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:43:45.425380 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:45.425387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:45.425584 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:45.426259 (+ 675us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:45.426433 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.427522 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.429143 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.430029 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.431100 (+ 1071us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.431265 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:45.431324 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 14:43:45.431423 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260928 14:43:45.432603 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.425600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.1.201.1:42921)
0928 14:43:45.425944 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:45.425961 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:45.425982 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:43:45.426364 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:45.426378 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:45.426598 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:45.426739 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:45.427134 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.428211 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.429710 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.430535 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.430631 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.430759 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:45.431596 (+ 837us) server_negotiation.cc:299] Negotiation successful
0928 14:43:45.431686 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260928 14:43:45.432878 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.427090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.1.201.3:41945)
0928 14:43:45.427429 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:45.427439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:45.427780 (+ 341us) server_negotiation.cc:407] Connection header received
0928 14:43:45.427877 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:45.427882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:45.428035 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:45.428149 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:45.428585 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.429681 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.431234 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.432068 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.432140 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.432261 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:45.432610 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 14:43:45.432732 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":151,"threads_started":1}
I20260928 14:43:45.433318 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.427432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:41945 (local address 127.0.0.1:52802)
0928 14:43:45.427678 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:45.427690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:45.427784 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:45.428288 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:45.428290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:45.428309 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:45.428448 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.428454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.429849 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.429851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.431117 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.431125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.432358 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.432361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.432454 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.432479 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:43:45.433119 (+ 640us) client_negotiation.cc:240] Negotiation successful
0928 14:43:45.433193 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":226,"thread_start_us":79,"threads_started":1}
I20260928 14:43:45.435045 17404 tablet_service.cc:1467] Processing CreateTablet for tablet 4f79af4e909048989b3f8a229b7e7665 (DEFAULT_TABLE table=test-table [id=311191cd263b4cf3a6d3f3a8c68eac2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:45.435025 17538 tablet_service.cc:1467] Processing CreateTablet for tablet 4f79af4e909048989b3f8a229b7e7665 (DEFAULT_TABLE table=test-table [id=311191cd263b4cf3a6d3f3a8c68eac2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:45.435984 17404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f79af4e909048989b3f8a229b7e7665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:45.436067 17538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f79af4e909048989b3f8a229b7e7665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:45.435889 17674 tablet_service.cc:1467] Processing CreateTablet for tablet 4f79af4e909048989b3f8a229b7e7665 (DEFAULT_TABLE table=test-table [id=311191cd263b4cf3a6d3f3a8c68eac2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:43:45.436789 17674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f79af4e909048989b3f8a229b7e7665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:43:45.443075 17758 tablet_bootstrap.cc:492] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Bootstrap starting.
I20260928 14:43:45.443854 17760 tablet_bootstrap.cc:492] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: Bootstrap starting.
I20260928 14:43:45.446178 17758 tablet_bootstrap.cc:654] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:45.446179 17760 tablet_bootstrap.cc:654] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:45.447554 17758 log.cc:824] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:45.447554 17760 log.cc:824] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:45.448215 17759 tablet_bootstrap.cc:492] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: Bootstrap starting.
I20260928 14:43:45.450714 17759 tablet_bootstrap.cc:654] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: Neither blocks nor log segments found. Creating new log.
I20260928 14:43:45.452116 17760 tablet_bootstrap.cc:492] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: No bootstrap required, opened a new log
I20260928 14:43:45.452363 17760 ts_tablet_manager.cc:1397] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 14:43:45.454696 17758 tablet_bootstrap.cc:492] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: No bootstrap required, opened a new log
I20260928 14:43:45.454878 17758 ts_tablet_manager.cc:1397] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260928 14:43:45.455693 17759 log.cc:824] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:43:45.457598 17760 raft_consensus.cc:348] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:43:45.457898 17760 raft_consensus.cc:374] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:45.457955 17760 raft_consensus.cc:729] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4908a7b69cd2411ba7e69b6785aeddb7, State: Initialized, Role: FOLLOWER
I20260928 14:43:45.458418 17760 consensus_queue.cc:260] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:43:45.458598 17758 raft_consensus.cc:348] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:43:45.458847 17758 raft_consensus.cc:374] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:45.458899 17758 raft_consensus.cc:729] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cac3ddab4ff465bb0fe08a49e6bfcaf, State: Initialized, Role: FOLLOWER
I20260928 14:43:45.459255 17740 heartbeater.cc:503] Master 127.1.201.62:34987 was elected leader, sending a full tablet report...
I20260928 14:43:45.459399 17760 ts_tablet_manager.cc:1428] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260928 14:43:45.459326 17758 consensus_queue.cc:260] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:43:45.459934 17470 heartbeater.cc:503] Master 127.1.201.62:34987 was elected leader, sending a full tablet report...
I20260928 14:43:45.461027 17759 tablet_bootstrap.cc:492] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: No bootstrap required, opened a new log
I20260928 14:43:45.461201 17759 ts_tablet_manager.cc:1397] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260928 14:43:45.461212 17758 ts_tablet_manager.cc:1428] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:43:45.465860 17759 raft_consensus.cc:348] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:43:45.466162 17759 raft_consensus.cc:374] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:43:45.466238 17759 raft_consensus.cc:729] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444a46ca7b14413da09a9027603aa7a2, State: Initialized, Role: FOLLOWER
I20260928 14:43:45.466724 17759 consensus_queue.cc:260] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:43:45.467707 17759 ts_tablet_manager.cc:1428] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:43:45.468060 17605 heartbeater.cc:503] Master 127.1.201.62:34987 was elected leader, sending a full tablet report...
I20260928 14:43:45.528750 17765 raft_consensus.cc:482] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:43:45.529068 17765 raft_consensus.cc:504] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:43:45.530318 17765 leader_election.cc:288] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771), 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945)
I20260928 14:43:45.534646 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.531179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:55353 (local address 127.1.201.3:41945)
0928 14:43:45.531338 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:45.531342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:45.531355 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:43:45.531487 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:45.531489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:45.531528 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:45.531588 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:45.531881 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.532451 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.533593 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.534073 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.534190 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.534253 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:45.534441 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 14:43:45.534517 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 14:43:45.534670 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.530684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:58855 (local address 127.1.201.2:35771)
0928 14:43:45.530836 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:45.530840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:45.530988 (+ 148us) server_negotiation.cc:407] Connection header received
0928 14:43:45.531321 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:45.531327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:45.531385 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:45.531463 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:43:45.531990 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.532483 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.533627 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.534092 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.534224 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.534289 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:45.534494 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 14:43:45.534566 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:43:45.534655 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.531005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:41945 (local address 127.1.201.1:55353)
0928 14:43:45.531289 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:45.531302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:45.531390 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:45.531627 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:45.531629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:45.531643 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:45.531764 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.531769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.532562 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.532566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.533440 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.533449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.534191 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.534195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.534292 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.534329 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:43:45.534397 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:43:45.534488 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":23,"thread_start_us":105,"threads_started":1}
I20260928 14:43:45.534675 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.530601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:35771 (local address 127.1.201.1:58855)
0928 14:43:45.530884 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:45.530900 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:43:45.531007 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:43:45.531691 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:43:45.531694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:43:45.531710 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:43:45.531863 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.531870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.532614 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.532616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.533447 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:43:45.533456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.534233 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.534236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:43:45.534346 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.534364 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:43:45.534459 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:43:45.534523 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":41,"thread_start_us":119,"threads_started":1}
I20260928 14:43:45.535507 17694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f79af4e909048989b3f8a229b7e7665" candidate_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" is_pre_election: true
I20260928 14:43:45.535511 17559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f79af4e909048989b3f8a229b7e7665" candidate_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444a46ca7b14413da09a9027603aa7a2" is_pre_election: true
I20260928 14:43:45.535863 17694 raft_consensus.cc:2454] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cac3ddab4ff465bb0fe08a49e6bfcaf in term 0.
I20260928 14:43:45.535863 17559 raft_consensus.cc:2454] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cac3ddab4ff465bb0fe08a49e6bfcaf in term 0.
I20260928 14:43:45.536355 17358 leader_election.cc:302] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4908a7b69cd2411ba7e69b6785aeddb7, 4cac3ddab4ff465bb0fe08a49e6bfcaf; no voters:
I20260928 14:43:45.536608 17765 raft_consensus.cc:2781] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:43:45.536693 17765 raft_consensus.cc:482] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:43:45.536739 17765 raft_consensus.cc:3037] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:45.537910 17765 raft_consensus.cc:504] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:43:45.538237 17765 leader_election.cc:288] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 1 election: Requested vote from peers 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771), 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945)
I20260928 14:43:45.538635 17559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f79af4e909048989b3f8a229b7e7665" candidate_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444a46ca7b14413da09a9027603aa7a2"
I20260928 14:43:45.538654 17694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f79af4e909048989b3f8a229b7e7665" candidate_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4908a7b69cd2411ba7e69b6785aeddb7"
I20260928 14:43:45.538774 17694 raft_consensus.cc:3037] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:45.538774 17559 raft_consensus.cc:3037] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:43:45.540020 17559 raft_consensus.cc:2454] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cac3ddab4ff465bb0fe08a49e6bfcaf in term 1.
I20260928 14:43:45.540020 17694 raft_consensus.cc:2454] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cac3ddab4ff465bb0fe08a49e6bfcaf in term 1.
I20260928 14:43:45.540338 17358 leader_election.cc:302] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4908a7b69cd2411ba7e69b6785aeddb7, 4cac3ddab4ff465bb0fe08a49e6bfcaf; no voters:
I20260928 14:43:45.540530 17765 raft_consensus.cc:2781] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:43:45.540772 17765 raft_consensus.cc:686] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 LEADER]: Becoming Leader. State: Replica: 4cac3ddab4ff465bb0fe08a49e6bfcaf, State: Running, Role: LEADER
I20260928 14:43:45.541123 17765 consensus_queue.cc:237] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:43:45.544464 17277 catalog_manager.cc:5495] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cac3ddab4ff465bb0fe08a49e6bfcaf (127.1.201.1). New cstate: current_term: 1 leader_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } health_report { overall_health: UNKNOWN } } }
I20260928 14:43:45.582228 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.578624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.1.201.1:42921)
0928 14:43:45.578770 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:43:45.578775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:43:45.578789 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:43:45.578949 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:43:45.578951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:43:45.579008 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:43:45.579114 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:43:45.579368 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.579862 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.580712 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:43:45.580930 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:43:45.581030 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:43:45.581350 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:43:45.581757 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:43:45.581941 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 14:43:45.582003 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 14:43:45.582079 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:43:45.588284 17559 raft_consensus.cc:1261] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2 [term 1 FOLLOWER]: Refusing update from remote peer 4cac3ddab4ff465bb0fe08a49e6bfcaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:43:45.588284 17694 raft_consensus.cc:1261] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 1 FOLLOWER]: Refusing update from remote peer 4cac3ddab4ff465bb0fe08a49e6bfcaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:43:45.589005 17765 consensus_queue.cc:1035] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:43:45.589195 17769 consensus_queue.cc:1035] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:43:45.595952 17772 mvcc.cc:205] Tried to move back new op lower bound from 7334324738402344960 to 7334324738219708416. Current Snapshot: MvccSnapshot[applied={T|T < 7334324738402344960}]
I20260928 14:43:45.598016 17774 mvcc.cc:205] Tried to move back new op lower bound from 7334324738402344960 to 7334324738219708416. Current Snapshot: MvccSnapshot[applied={T|T < 7334324738402344960}]
I20260928 14:43:45.605137 17278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:43:45.605319 17278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:43:45.607746 17278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Sending DeleteTablet for 3 replicas of tablet 4f79af4e909048989b3f8a229b7e7665
I20260928 14:43:45.608587 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 17340
I20260928 14:43:45.608590 17538 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f79af4e909048989b3f8a229b7e7665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:45 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:34966
I20260928 14:43:45.609074 17674 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f79af4e909048989b3f8a229b7e7665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:45 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:52802
I20260928 14:43:45.609512 17782 tablet_replica.cc:330] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: stopping tablet replica
I20260928 14:43:45.609509 17781 tablet_replica.cc:330] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: stopping tablet replica
I20260928 14:43:45.609828 17781 raft_consensus.cc:2229] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:43:45.609828 17782 raft_consensus.cc:2229] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:43:45.610127 17781 raft_consensus.cc:2258] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:43:45.610127 17782 raft_consensus.cc:2258] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:43:45.611354 17781 ts_tablet_manager.cc:1905] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:43:45.613427 17781 ts_tablet_manager.cc:1918] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:43:45.613538 17781 log.cc:1196] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/4f79af4e909048989b3f8a229b7e7665
I20260928 14:43:45.613960 17781 ts_tablet_manager.cc:1939] T 4f79af4e909048989b3f8a229b7e7665 P 444a46ca7b14413da09a9027603aa7a2: Deleting consensus metadata
I20260928 14:43:45.614838 17782 ts_tablet_manager.cc:1905] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:43:45.615342 17264 catalog_manager.cc:4841] TS 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771): tablet 4f79af4e909048989b3f8a229b7e7665 (table test-table [id=311191cd263b4cf3a6d3f3a8c68eac2a]) successfully deleted
W20260928 14:43:45.615737 17264 connection.cc:535] client connection to 127.1.201.1:42921 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 14:43:45.615878 17264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:43:45.616034 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 17475
I20260928 14:43:45.618220 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:43:45.617962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42921 (local address 127.0.0.1:34300)
0928 14:43:45.618088 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:43:45.618153 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:42921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 14:43:45.618419 17264 catalog_manager.cc:4584] TS 4cac3ddab4ff465bb0fe08a49e6bfcaf (127.1.201.1:42921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f79af4e909048989b3f8a229b7e7665: Network error: Client connection negotiation failed: client connection to 127.1.201.1:42921: connect: Connection refused (error 111)
I20260928 14:43:45.618445 17782 ts_tablet_manager.cc:1918] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:43:45.618541 17782 log.cc:1196] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/4f79af4e909048989b3f8a229b7e7665
I20260928 14:43:45.618901 17782 ts_tablet_manager.cc:1939] T 4f79af4e909048989b3f8a229b7e7665 P 4908a7b69cd2411ba7e69b6785aeddb7: Deleting consensus metadata
I20260928 14:43:45.620122 17264 catalog_manager.cc:4841] TS 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945): tablet 4f79af4e909048989b3f8a229b7e7665 (table test-table [id=311191cd263b4cf3a6d3f3a8c68eac2a]) successfully deleted
I20260928 14:43:45.622736 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 17610
I20260928 14:43:45.631398 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 17246
I20260928 14:44:05.638237 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:34987
--webserver_interface=127.1.201.62
--webserver_port=35771
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:34987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:05.749092 17784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:05.749343 17784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:05.749382 17784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:05.753330 17784 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:44:05.753384 17784 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:44:05.753400 17784 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:44:05.753414 17784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:05.753432 17784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:05.753449 17784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:05.753465 17784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:05.753480 17784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:05.758638 17784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:34987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=35771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:05.759572 17784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:05.760627 17784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:05.773329 17791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:05.773343 17793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:05.773415 17784 server_base.cc:1034] running on GCE node
W20260928 14:44:05.773344 17790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:05.774315 17784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:05.775494 17784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:05.776686 17784 hybrid_clock.cc:648] HybridClock initialized: now 1790606645776668 us; error 32 us; skew 500 ppm
Sep 28 14:44:05 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606645, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:05.779590 17784 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:44:05.780548 17784 webserver.cc:460] Webserver started at http://127.1.201.62:35771/ using document root <none> and password file <none>
I20260928 14:44:05.781139 17784 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:05.781245 17784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:05.785104 17784 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:05.787318 17800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:05.788511 17784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:44:05.788651 17784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab"
format_stamp: "Formatted at 2026-09-28 14:43:44 on dist-test-slave-0n27"
server_key: "f823842a0f54d7efccec6e2385dea35a"
server_key_iv: "d0d4a4e7f74b50c73ec9b91239353c67"
server_key_version: "encryptionkey@0"
I20260928 14:44:05.789263 17784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:05.813843 17784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:05.816712 17784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:05.816901 17784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:05.823823 17784 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:34987
I20260928 14:44:05.824115 17852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:34987 every 8 connection(s)
I20260928 14:44:05.824928 17784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:05.829259 17853 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:44:05.830943 17853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Bootstrap starting.
I20260928 14:44:05.835181 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 17784
I20260928 14:44:05.835625 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:42921
--local_ip_for_outbound_sockets=127.1.201.1
--tserver_master_addrs=127.1.201.62:34987
--webserver_port=45681
--webserver_interface=127.1.201.1
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:44:05.847870 17853 log.cc:824] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:05.856282 17853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:44:05.856630 17853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Bootstrap complete.
I20260928 14:44:05.860540 17853 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:44:05.860922 17853 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca3d307a25a4e14a42e6d6d57a7b1ab, State: Initialized, Role: FOLLOWER
I20260928 14:44:05.861378 17853 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:44:05.861515 17853 raft_consensus.cc:388] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:05.861595 17853 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:05.861701 17853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:44:05.863474 17853 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:44:05.863812 17853 leader_election.cc:302] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca3d307a25a4e14a42e6d6d57a7b1ab; no voters:
I20260928 14:44:05.864099 17853 leader_election.cc:288] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:44:05.864235 17858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:44:05.864497 17858 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 2 LEADER]: Becoming Leader. State: Replica: bca3d307a25a4e14a42e6d6d57a7b1ab, State: Running, Role: LEADER
I20260928 14:44:05.864873 17858 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:44:05.865290 17853 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:05.866266 17860 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca3d307a25a4e14a42e6d6d57a7b1ab. Latest consensus state: current_term: 2 leader_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } } }
I20260928 14:44:05.866379 17860 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:05.867002 17859 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } } }
I20260928 14:44:05.867076 17859 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:05.869971 17875 catalog_manager.cc:1259] Loaded cluster ID: 82ec9cb8498146019336476102339e3e
I20260928 14:44:05.870028 17875 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: loading cluster ID for follower catalog manager: success
I20260928 14:44:05.872392 17875 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: acquiring CA information for follower catalog manager: success
I20260928 14:44:05.873889 17875 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:44:05.874045 17867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:05.876328 17867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:05.877199 17867 catalog_manager.cc:1259] Loaded cluster ID: 82ec9cb8498146019336476102339e3e
I20260928 14:44:05.877255 17867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:05.877923 17867 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:05.878418 17867 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Loaded TSK: 0
I20260928 14:44:05.899245 17867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Generated new TSK 1
I20260928 14:44:05.899483 17867 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:44:05.947793 17856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:05.948101 17856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:05.948175 17856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:05.952293 17856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:05.952363 17856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:05.952447 17856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:44:05.957563 17856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:42921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=45681
--tserver_master_addrs=127.1.201.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:05.958561 17856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:05.959551 17856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:05.969893 17885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:05.969949 17883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:05.969902 17882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:05.970253 17856 server_base.cc:1034] running on GCE node
I20260928 14:44:05.970685 17856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:05.971298 17856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:05.972484 17856 hybrid_clock.cc:648] HybridClock initialized: now 1790606645972459 us; error 45 us; skew 500 ppm
Sep 28 14:44:05 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606645, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:05.975370 17856 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:44:05.976434 17856 webserver.cc:460] Webserver started at http://127.1.201.1:45681/ using document root <none> and password file <none>
I20260928 14:44:05.977075 17856 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:05.977170 17856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:05.982043 17856 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:44:05.984628 17892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:05.986101 17856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:44:05.986230 17856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf"
format_stamp: "Formatted at 2026-09-28 14:43:44 on dist-test-slave-0n27"
server_key: "5203a68597435e3531724e9c733bcda7"
server_key_iv: "dfce6399cebc7ff523f5854686c00f8e"
server_key_version: "encryptionkey@0"
I20260928 14:44:05.986907 17856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:06.021639 17856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:06.025346 17856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:06.025528 17856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:06.026222 17856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:06.027547 17899 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:44:06.032292 17856 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:44:06.032379 17856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260928 14:44:06.032464 17856 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:44:06.034328 17856 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:44:06.034394 17856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:06.034688 17899 tablet_bootstrap.cc:492] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Bootstrap starting.
I20260928 14:44:06.044749 17856 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:42921
I20260928 14:44:06.044816 18007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:42921 every 8 connection(s)
I20260928 14:44:06.045706 17856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:44:06.052568 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 17856
I20260928 14:44:06.052855 17899 log.cc:824] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:06.053072 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:35771
--local_ip_for_outbound_sockets=127.1.201.2
--tserver_master_addrs=127.1.201.62:34987
--webserver_port=33379
--webserver_interface=127.1.201.2
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:44:06 dist-test-slave-0n27 krb5kdc[17232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606645, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:44:06.060771 17899 tablet_bootstrap.cc:492] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:44:06.061138 17899 tablet_bootstrap.cc:492] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Bootstrap complete.
I20260928 14:44:06.062402 17899 ts_tablet_manager.cc:1397] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.023s
I20260928 14:44:06.064946 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.048089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:39649 (local address 127.1.201.62:34987)
0928 14:44:06.048463 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.048482 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.049185 (+ 703us) server_negotiation.cc:407] Connection header received
0928 14:44:06.050337 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.050348 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.050622 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.050932 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:06.051982 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.053418 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.054844 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.055554 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.055758 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.058797 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:06.058805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:06.058826 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:06.058865 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:06.061140 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.061894 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.061902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.061911 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.062063 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.062484 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.062488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.062490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.062923 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:06.063051 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.063760 (+ 709us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.064079 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":115,"threads_started":1}
I20260928 14:44:06.065403 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.048066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:39649)
0928 14:44:06.049023 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:06.049063 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:06.049943 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:06.050980 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:06.050987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:06.051385 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:06.051822 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.051835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.053608 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.053614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.054690 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.054705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.055736 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.055744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.055975 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.056681 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:06.056706 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:06.058573 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:06.061303 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:06.061310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:06.061324 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:06.061740 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:06.062165 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:06.062170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:06.062174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:06.062362 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:06.063053 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:06.063062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:06.063389 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 14:44:06.063601 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 14:44:06.064026 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":140,"threads_started":1}
I20260928 14:44:06.065109 17899 raft_consensus.cc:348] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:44:06.065606 17899 raft_consensus.cc:729] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cac3ddab4ff465bb0fe08a49e6bfcaf, State: Initialized, Role: FOLLOWER
I20260928 14:44:06.066079 17899 consensus_queue.cc:260] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } }
I20260928 14:44:06.067101 17899 ts_tablet_manager.cc:1428] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:44:06.067909 18008 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34987
I20260928 14:44:06.068140 18008 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:06.068637 18008 heartbeater.cc:511] Master 127.1.201.62:34987 requested a full tablet report, sending...
I20260928 14:44:06.071087 17817 ts_manager.cc:194] Registered new tserver with Master: 4cac3ddab4ff465bb0fe08a49e6bfcaf (127.1.201.1:42921)
I20260928 14:44:06.074342 17817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:39649
I20260928 14:44:06.076932 18008 heartbeater.cc:503] Master 127.1.201.62:34987 was elected leader, sending a full tablet report...
Sep 28 14:44:06 dist-test-slave-0n27 krb5kdc[17232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606645, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.1@KRBTEST.COM
I20260928 14:44:06.085691 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.073605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42921 (local address 127.0.0.1:58284)
0928 14:44:06.074333 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:06.074378 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:06.074558 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:06.075167 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:06.075172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:06.077430 (+ 2258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:06.077584 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.077593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.078903 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.078906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.079586 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.079595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.080254 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.080256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.080329 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.080836 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:06.080847 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:06.082139 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:06.084105 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:06.084113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:06.084117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:06.084338 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:06.084670 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:06.084672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:06.084673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:06.084754 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:06.085221 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:06.085225 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:06.085315 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 14:44:06.085417 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:44:06.085553 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":101,"threads_started":1}
I20260928 14:44:06.085835 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.073694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.1.201.1:42921)
0928 14:44:06.074067 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.074078 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.074457 (+ 379us) server_negotiation.cc:407] Connection header received
0928 14:44:06.074721 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.074726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.074914 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.075055 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:06.077746 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.078800 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.079700 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.080152 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.080257 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.082268 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:06.082277 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:06.082289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:06.082314 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:06.083942 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.084459 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.084465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.084466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.084555 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.084840 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.084843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.084845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.085107 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:06.085247 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.085508 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.085669 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":99,"threads_started":1}
I20260928 14:44:06.086694 17941 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f79af4e909048989b3f8a229b7e7665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:43:45 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260928 14:44:06.087471 18017 tablet_replica.cc:330] stopping tablet replica
I20260928 14:44:06.087749 18017 raft_consensus.cc:2229] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:06.087976 18017 raft_consensus.cc:2258] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:06.088855 18017 ts_tablet_manager.cc:1905] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:06.092324 18017 ts_tablet_manager.cc:1918] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:06.092419 18017 log.cc:1196] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal/wals/4f79af4e909048989b3f8a229b7e7665
I20260928 14:44:06.092691 18017 ts_tablet_manager.cc:1939] T 4f79af4e909048989b3f8a229b7e7665 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Deleting consensus metadata
I20260928 14:44:06.094449 17803 catalog_manager.cc:4841] TS 4cac3ddab4ff465bb0fe08a49e6bfcaf (127.1.201.1:42921): tablet 4f79af4e909048989b3f8a229b7e7665 (table test-table [id=311191cd263b4cf3a6d3f3a8c68eac2a]) successfully deleted
W20260928 14:44:06.169045 18013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:06.169292 18013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:06.169355 18013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:06.173194 18013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:06.173272 18013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:06.173352 18013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:44:06.178500 18013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:35771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=33379
--tserver_master_addrs=127.1.201.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:06.179556 18013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:06.180603 18013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:06.190232 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:06.190239 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:06.190274 18013 server_base.cc:1034] running on GCE node
W20260928 14:44:06.190240 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:06.190835 18013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:06.191500 18013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:06.192696 18013 hybrid_clock.cc:648] HybridClock initialized: now 1790606646192672 us; error 40 us; skew 500 ppm
Sep 28 14:44:06 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606646, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:06.195421 18013 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:44:06.196390 18013 webserver.cc:460] Webserver started at http://127.1.201.2:33379/ using document root <none> and password file <none>
I20260928 14:44:06.197001 18013 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:06.197103 18013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:06.200889 18013 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:06.202865 18032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:06.204030 18013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:44:06.204155 18013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "444a46ca7b14413da09a9027603aa7a2"
format_stamp: "Formatted at 2026-09-28 14:43:45 on dist-test-slave-0n27"
server_key: "7ae12581a3d0204e11d56108346bbf0f"
server_key_iv: "4fc5bb64675b24bca9db72d5f5f0984c"
server_key_version: "encryptionkey@0"
I20260928 14:44:06.204680 18013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:06.219668 18013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:06.222293 18013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:06.222494 18013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:06.223043 18013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:06.223820 18013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:06.223886 18013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:06.223969 18013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:06.224013 18013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:06.232375 18013 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:35771
I20260928 14:44:06.232402 18145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:35771 every 8 connection(s)
I20260928 14:44:06.233304 18013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:44:06.239948 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 18013
I20260928 14:44:06.240463 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:41945
--local_ip_for_outbound_sockets=127.1.201.3
--tserver_master_addrs=127.1.201.62:34987
--webserver_port=41459
--webserver_interface=127.1.201.3
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:44:06 dist-test-slave-0n27 krb5kdc[17232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606646, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:44:06.246137 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.234956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:33717 (local address 127.1.201.62:34987)
0928 14:44:06.235121 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.235126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.235928 (+ 802us) server_negotiation.cc:407] Connection header received
0928 14:44:06.237144 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.237151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.237198 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.237271 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:06.238442 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.238956 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.240037 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.240406 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.240513 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.242820 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:06.242824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:06.242826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:06.242850 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:06.244153 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.244650 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.244654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.244666 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.244738 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.245054 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.245059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.245061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.245246 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:06.245344 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.245815 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.245950 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:44:06.246872 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.235362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:33717)
0928 14:44:06.235785 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:06.235825 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:06.236962 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:06.237427 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:06.237432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:06.237762 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:06.238317 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.238331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.239057 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.239059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.239911 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.239920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.240572 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.240579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.240762 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.241322 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:06.241337 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:06.242662 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:06.244269 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:06.244274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:06.244288 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:06.244537 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:06.244826 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:06.244831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:06.244833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:06.244964 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:06.245341 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:06.245348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:06.245572 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 14:44:06.245744 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 14:44:06.246055 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
I20260928 14:44:06.247994 18146 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34987
I20260928 14:44:06.248252 18146 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:06.248858 18146 heartbeater.cc:511] Master 127.1.201.62:34987 requested a full tablet report, sending...
I20260928 14:44:06.249995 17817 ts_manager.cc:194] Registered new tserver with Master: 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771)
I20260928 14:44:06.250849 17817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:33717
W20260928 14:44:06.349828 18149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:06.350062 18149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:06.350102 18149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:06.353998 18149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:06.354054 18149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:06.354111 18149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:44:06.359387 18149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:41945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=41459
--tserver_master_addrs=127.1.201.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:06.360414 18149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:06.361377 18149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:06.370839 18156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:06.370867 18158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:06.370867 18155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:06.370963 18149 server_base.cc:1034] running on GCE node
I20260928 14:44:06.371382 18149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:06.371984 18149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:06.373157 18149 hybrid_clock.cc:648] HybridClock initialized: now 1790606646373129 us; error 37 us; skew 500 ppm
Sep 28 14:44:06 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606646, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:06.375802 18149 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:44:06.376662 18149 webserver.cc:460] Webserver started at http://127.1.201.3:41459/ using document root <none> and password file <none>
I20260928 14:44:06.377213 18149 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:06.377321 18149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:06.381096 18149 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:06.383044 18165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:06.384188 18149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:44:06.384310 18149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "4908a7b69cd2411ba7e69b6785aeddb7"
format_stamp: "Formatted at 2026-09-28 14:43:45 on dist-test-slave-0n27"
server_key: "76d1449e594f6b0dee29a967c5477e00"
server_key_iv: "ef3821166a470226f96c840fb7ae02d8"
server_key_version: "encryptionkey@0"
I20260928 14:44:06.384835 18149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:06.407552 18149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:06.410698 18149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:06.410892 18149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:06.411470 18149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:06.412253 18149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:06.412317 18149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:06.412384 18149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:06.412421 18149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:06.421178 18149 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:41945
I20260928 14:44:06.421195 18278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:41945 every 8 connection(s)
I20260928 14:44:06.422194 18149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:44:06.427076 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 18149
Sep 28 14:44:06 dist-test-slave-0n27 krb5kdc[17232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606646, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:44:06.436724 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.427757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.1.201.62:34987)
0928 14:44:06.428072 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.428077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.428105 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:44:06.428243 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.428245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.428285 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.428360 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:06.429243 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.430141 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.431167 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.431475 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.431602 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.433282 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:06.433285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:06.433287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:06.433311 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:06.434952 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.435444 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.435448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.435450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.435539 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.435875 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.435877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.435878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.436038 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:06.436183 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.436466 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.436563 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":104,"threads_started":1}
I20260928 14:44:06.438025 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.423903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:55567 (local address 127.1.201.62:34987)
0928 14:44:06.424046 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.424051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.424906 (+ 855us) server_negotiation.cc:407] Connection header received
0928 14:44:06.426478 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.426481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.426531 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.426644 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:06.428114 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.428962 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.430628 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.430964 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.431081 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.434247 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:06.434252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:06.434256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:06.434298 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:06.435692 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.436248 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.436250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.436251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.436309 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.436747 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.436751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.436753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.437005 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:06.437085 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.437737 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.437874 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:44:06.438966 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.424347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:55567)
0928 14:44:06.424768 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:06.424809 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:06.426255 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:06.426803 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:06.426811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:06.427254 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:06.427932 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.427958 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.429114 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.429117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.430518 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.430527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.431161 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.431166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.431349 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.431991 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:06.432202 (+ 211us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:06.434073 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:06.435798 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:06.435804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:06.435823 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:06.436159 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:06.436404 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:06.436409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:06.436411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:06.436615 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:06.437159 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:06.437166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:06.437446 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 14:44:06.437678 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 14:44:06.438046 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":113,"threads_started":1}
I20260928 14:44:06.440048 18279 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34987
I20260928 14:44:06.440299 18279 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:06.440762 18279 heartbeater.cc:511] Master 127.1.201.62:34987 requested a full tablet report, sending...
I20260928 14:44:06.441834 17817 ts_manager.cc:194] Registered new tserver with Master: 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945)
I20260928 14:44:06.442384 17817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:55567
I20260928 14:44:06.450237 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:44:06.454267 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.450815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.1.201.62:34987)
0928 14:44:06.450952 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.450956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.451123 (+ 167us) server_negotiation.cc:407] Connection header received
0928 14:44:06.451293 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.451295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.451380 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.451475 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:44:06.451716 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.452200 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.453089 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.453309 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.453416 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.453715 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:44:06.453884 (+ 169us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 14:44:06.454143 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37276: token expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 14:44:06.454341 18010 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37276: token expired
W20260928 14:44:06.454679 17743 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.201.62:34987)
I20260928 14:44:06.462186 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.455043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.1.201.62:34987)
0928 14:44:06.455174 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.455180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.455193 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:44:06.455302 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.455305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.455340 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.455418 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:06.456011 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.456474 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.457651 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.457945 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.458031 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.459179 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:06.459182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:06.459184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:06.459206 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:06.460716 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.461097 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.461101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.461103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.461175 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:06.461391 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:06.461393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:06.461394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:06.461549 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:06.461677 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.461905 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.461995 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:44:06.464562 17817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:44:06.468142 17817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:44:06.475653 17941 tablet_service.cc:1467] Processing CreateTablet for tablet b47f6b1873424b68bb54ce4329ceb8d4 (DEFAULT_TABLE table=test-table [id=400c37eabe714678bd7f91f2c22cf613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:06.476296 17941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47f6b1873424b68bb54ce4329ceb8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:06.478922 17899 tablet_bootstrap.cc:492] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Bootstrap starting.
I20260928 14:44:06.479532 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.474386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:35771 (local address 127.0.0.1:35102)
0928 14:44:06.474554 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:06.474569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:06.474640 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:06.475663 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:06.475666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:06.475684 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:06.475790 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.475797 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.477009 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.477012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.478127 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.478137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.479220 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.479223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.479308 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.479342 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 14:44:06.479382 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 14:44:06.479442 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 14:44:06.479688 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.474682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:41945 (local address 127.0.0.1:42660)
0928 14:44:06.474959 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:06.474975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:06.475057 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:06.475789 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:06.475792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:06.475810 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:06.475929 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.475936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.477324 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.477328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.478353 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.478360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.479242 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.479245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.479348 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.479374 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:44:06.479423 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:44:06.479490 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":108,"threads_started":1}
I20260928 14:44:06.479844 17899 tablet_bootstrap.cc:654] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:06.480064 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.474929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.1.201.3:41945)
0928 14:44:06.475316 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.475327 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.475348 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:44:06.475417 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.475422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.475630 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.476068 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:06.476240 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.477210 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.478456 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.479105 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.479601 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.479758 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.479816 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.479909 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260928 14:44:06.480258 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.474763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.1.201.2:35771)
0928 14:44:06.475243 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.475254 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.475279 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:44:06.475344 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.475347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.475506 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.475682 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:06.475927 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.476890 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.478262 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.479074 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.479855 (+ 781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.479959 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.479997 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.480092 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":131,"threads_started":1}
I20260928 14:44:06.481709 17899 tablet_bootstrap.cc:492] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: No bootstrap required, opened a new log
I20260928 14:44:06.481838 17899 ts_tablet_manager.cc:1397] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 14:44:06.482288 17899 raft_consensus.cc:348] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:06.482412 17899 raft_consensus.cc:374] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:06.482448 17899 raft_consensus.cc:729] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cac3ddab4ff465bb0fe08a49e6bfcaf, State: Initialized, Role: FOLLOWER
I20260928 14:44:06.482614 17899 consensus_queue.cc:260] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:06.482877 17899 ts_tablet_manager.cc:1428] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:44:06.483064 18213 tablet_service.cc:1467] Processing CreateTablet for tablet b47f6b1873424b68bb54ce4329ceb8d4 (DEFAULT_TABLE table=test-table [id=400c37eabe714678bd7f91f2c22cf613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:06.483078 18080 tablet_service.cc:1467] Processing CreateTablet for tablet b47f6b1873424b68bb54ce4329ceb8d4 (DEFAULT_TABLE table=test-table [id=400c37eabe714678bd7f91f2c22cf613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:06.484062 18080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47f6b1873424b68bb54ce4329ceb8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:06.484268 18213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47f6b1873424b68bb54ce4329ceb8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:06.490619 18292 tablet_bootstrap.cc:492] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: Bootstrap starting.
I20260928 14:44:06.491158 18293 tablet_bootstrap.cc:492] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: Bootstrap starting.
I20260928 14:44:06.493076 18292 tablet_bootstrap.cc:654] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:06.493075 18293 tablet_bootstrap.cc:654] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:06.493916 18292 log.cc:824] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:06.494182 18293 log.cc:824] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:06.495816 18292 tablet_bootstrap.cc:492] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: No bootstrap required, opened a new log
I20260928 14:44:06.495942 18293 tablet_bootstrap.cc:492] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: No bootstrap required, opened a new log
I20260928 14:44:06.495998 18292 ts_tablet_manager.cc:1397] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:44:06.496129 18293 ts_tablet_manager.cc:1397] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:44:06.498955 18293 raft_consensus.cc:348] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:06.498955 18292 raft_consensus.cc:348] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:06.499178 18292 raft_consensus.cc:374] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:06.499203 18293 raft_consensus.cc:374] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:06.499236 18292 raft_consensus.cc:729] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444a46ca7b14413da09a9027603aa7a2, State: Initialized, Role: FOLLOWER
I20260928 14:44:06.499274 18293 raft_consensus.cc:729] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4908a7b69cd2411ba7e69b6785aeddb7, State: Initialized, Role: FOLLOWER
I20260928 14:44:06.499656 18292 consensus_queue.cc:260] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:06.499665 18293 consensus_queue.cc:260] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:06.500535 18293 ts_tablet_manager.cc:1428] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 14:44:06.500540 18292 ts_tablet_manager.cc:1428] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 14:44:06.500587 18279 heartbeater.cc:503] Master 127.1.201.62:34987 was elected leader, sending a full tablet report...
I20260928 14:44:06.501550 18146 heartbeater.cc:503] Master 127.1.201.62:34987 was elected leader, sending a full tablet report...
W20260928 14:44:06.546881 18009 tablet.cc:2369] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:06.627290 18297 raft_consensus.cc:482] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:06.627506 18297 raft_consensus.cc:504] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:06.628734 18297 leader_election.cc:288] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771), 4cac3ddab4ff465bb0fe08a49e6bfcaf (127.1.201.1:42921)
I20260928 14:44:06.632769 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.628987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:35771 (local address 127.1.201.3:43351)
0928 14:44:06.629124 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:06.629152 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:06.629243 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:06.629491 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:06.629494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:06.629517 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:06.629640 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.629646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.630625 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.630627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.631720 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.631729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.632459 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.632461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.632538 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.632557 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:44:06.632613 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:44:06.632672 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 14:44:06.632861 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.629104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:43351 (local address 127.1.201.2:35771)
0928 14:44:06.629246 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.629251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.629266 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:44:06.629335 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.629340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.629390 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.629461 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:06.629763 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.630532 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.631834 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.632354 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.632454 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.632516 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.632653 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.632721 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:44:06.633296 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.629303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42921 (local address 127.1.201.3:53119)
0928 14:44:06.629715 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:06.629729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:06.629844 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:06.630108 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:06.630110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:06.630122 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:06.630214 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.630217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.630991 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.630994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.631996 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:06.632003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.632871 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.632875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:06.632982 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.633015 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:44:06.633086 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:44:06.633178 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":120,"threads_started":1}
I20260928 14:44:06.633515 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.629440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:53119 (local address 127.1.201.1:42921)
0928 14:44:06.629673 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.629677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.629804 (+ 127us) server_negotiation.cc:407] Connection header received
0928 14:44:06.629924 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.629927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.629990 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.630096 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:06.630312 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.630889 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.632099 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.632736 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.632832 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.633003 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.633196 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.633275 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260928 14:44:06.633489 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47f6b1873424b68bb54ce4329ceb8d4" candidate_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444a46ca7b14413da09a9027603aa7a2" is_pre_election: true
I20260928 14:44:06.633822 18100 raft_consensus.cc:2454] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4908a7b69cd2411ba7e69b6785aeddb7 in term 0.
I20260928 14:44:06.634063 17961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47f6b1873424b68bb54ce4329ceb8d4" candidate_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" is_pre_election: true
I20260928 14:44:06.634352 17961 raft_consensus.cc:2454] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4908a7b69cd2411ba7e69b6785aeddb7 in term 0.
I20260928 14:44:06.634331 18168 leader_election.cc:302] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 444a46ca7b14413da09a9027603aa7a2, 4908a7b69cd2411ba7e69b6785aeddb7; no voters:
I20260928 14:44:06.634637 18297 raft_consensus.cc:2781] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:44:06.634760 18297 raft_consensus.cc:482] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:44:06.634824 18297 raft_consensus.cc:3037] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:06.635836 18297 raft_consensus.cc:504] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:06.636158 18297 leader_election.cc:288] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [CANDIDATE]: Term 1 election: Requested vote from peers 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771), 4cac3ddab4ff465bb0fe08a49e6bfcaf (127.1.201.1:42921)
I20260928 14:44:06.636487 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47f6b1873424b68bb54ce4329ceb8d4" candidate_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444a46ca7b14413da09a9027603aa7a2"
I20260928 14:44:06.636552 17961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47f6b1873424b68bb54ce4329ceb8d4" candidate_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf"
I20260928 14:44:06.636617 18100 raft_consensus.cc:3037] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:06.636680 17961 raft_consensus.cc:3037] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:06.637584 18100 raft_consensus.cc:2454] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4908a7b69cd2411ba7e69b6785aeddb7 in term 1.
I20260928 14:44:06.637639 17961 raft_consensus.cc:2454] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4908a7b69cd2411ba7e69b6785aeddb7 in term 1.
I20260928 14:44:06.637907 18168 leader_election.cc:302] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 444a46ca7b14413da09a9027603aa7a2, 4908a7b69cd2411ba7e69b6785aeddb7; no voters:
I20260928 14:44:06.638064 18297 raft_consensus.cc:2781] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:06.638279 18297 raft_consensus.cc:686] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 1 LEADER]: Becoming Leader. State: Replica: 4908a7b69cd2411ba7e69b6785aeddb7, State: Running, Role: LEADER
I20260928 14:44:06.638585 18297 consensus_queue.cc:237] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:06.641592 17816 catalog_manager.cc:5495] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3). New cstate: current_term: 1 leader_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } health_report { overall_health: UNKNOWN } } }
I20260928 14:44:06.671846 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.668285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.1.201.3:41945)
0928 14:44:06.668434 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:06.668439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:06.668453 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:44:06.668609 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:06.668612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:06.668672 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:06.668780 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:44:06.669009 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.669498 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.670309 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:06.670518 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:06.670622 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:06.670915 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:44:06.671408 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:44:06.671531 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:06.671622 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:44:06.671683 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 14:44:06.673087 18280 tablet.cc:2369] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:06.677500 17961 raft_consensus.cc:1261] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Refusing update from remote peer 4908a7b69cd2411ba7e69b6785aeddb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:06.677505 18100 raft_consensus.cc:1261] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2 [term 1 FOLLOWER]: Refusing update from remote peer 4908a7b69cd2411ba7e69b6785aeddb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:06.678177 18297 consensus_queue.cc:1035] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:06.678323 18300 consensus_queue.cc:1035] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:06.684762 18304 mvcc.cc:205] Tried to move back new op lower bound from 7334324824784285696 to 7334324824634294272. Current Snapshot: MvccSnapshot[applied={T|T < 7334324824784285696}]
I20260928 14:44:06.685312 18305 mvcc.cc:205] Tried to move back new op lower bound from 7334324824784285696 to 7334324824634294272. Current Snapshot: MvccSnapshot[applied={T|T < 7334324824784285696}]
I20260928 14:44:06.693581 17815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:06.693729 17815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:06.696034 17815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Sending DeleteTablet for 3 replicas of tablet b47f6b1873424b68bb54ce4329ceb8d4
I20260928 14:44:06.696735 17941 tablet_service.cc:1514] Processing DeleteTablet for tablet b47f6b1873424b68bb54ce4329ceb8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:06 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260928 14:44:06.696828 18080 tablet_service.cc:1514] Processing DeleteTablet for tablet b47f6b1873424b68bb54ce4329ceb8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:06 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:35102
I20260928 14:44:06.697150 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 17856
I20260928 14:44:06.697319 18213 tablet_service.cc:1514] Processing DeleteTablet for tablet b47f6b1873424b68bb54ce4329ceb8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:06 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:42660
I20260928 14:44:06.698017 18314 tablet_replica.cc:330] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: stopping tablet replica
I20260928 14:44:06.698025 18313 tablet_replica.cc:330] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: stopping tablet replica
I20260928 14:44:06.698314 18314 raft_consensus.cc:2229] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:44:06.698472 18313 raft_consensus.cc:2229] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:06.698701 18314 raft_consensus.cc:2258] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:06.698850 18313 raft_consensus.cc:2258] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:06.700174 18314 ts_tablet_manager.cc:1905] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:06.700368 18313 ts_tablet_manager.cc:1905] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:06.702939 18314 ts_tablet_manager.cc:1918] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:06.703034 18314 log.cc:1196] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/b47f6b1873424b68bb54ce4329ceb8d4
I20260928 14:44:06.703354 18314 ts_tablet_manager.cc:1939] T b47f6b1873424b68bb54ce4329ceb8d4 P 4908a7b69cd2411ba7e69b6785aeddb7: Deleting consensus metadata
I20260928 14:44:06.703902 18313 ts_tablet_manager.cc:1918] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:06.704066 18313 log.cc:1196] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/b47f6b1873424b68bb54ce4329ceb8d4
I20260928 14:44:06.704458 18313 ts_tablet_manager.cc:1939] T b47f6b1873424b68bb54ce4329ceb8d4 P 444a46ca7b14413da09a9027603aa7a2: Deleting consensus metadata
I20260928 14:44:06.704816 17803 catalog_manager.cc:4841] TS 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945): tablet b47f6b1873424b68bb54ce4329ceb8d4 (table test-table [id=400c37eabe714678bd7f91f2c22cf613]) successfully deleted
I20260928 14:44:06.705706 17803 catalog_manager.cc:4841] TS 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771): tablet b47f6b1873424b68bb54ce4329ceb8d4 (table test-table [id=400c37eabe714678bd7f91f2c22cf613]) successfully deleted
W20260928 14:44:06.706156 17803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:42921 (error 108)
I20260928 14:44:06.706940 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 18013
I20260928 14:44:06.707151 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:06.706867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42921 (local address 127.0.0.1:58290)
0928 14:44:06.707023 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:06.707081 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:42921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260928 14:44:06.710271 17803 catalog_manager.cc:4584] TS 4cac3ddab4ff465bb0fe08a49e6bfcaf (127.1.201.1:42921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b47f6b1873424b68bb54ce4329ceb8d4: Network error: Client connection negotiation failed: client connection to 127.1.201.1:42921: connect: Connection refused (error 111)
I20260928 14:44:06.714268 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 18149
I20260928 14:44:06.719878 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 17784
I20260928 14:44:26.726413 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:34987
--webserver_interface=127.1.201.62
--webserver_port=35771
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:34987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:26.833619 18316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:26.833963 18316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:26.834074 18316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:26.838580 18316 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:44:26.838654 18316 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:44:26.838681 18316 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:44:26.838716 18316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:26.838750 18316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:26.838828 18316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:26.838862 18316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:26.838893 18316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:26.844061 18316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:34987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=35771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:26.845065 18316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:26.846140 18316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:26.854367 18324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:26.854379 18322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:26.854375 18321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:26.854668 18316 server_base.cc:1034] running on GCE node
I20260928 14:44:26.855291 18316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:26.856277 18316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:26.857434 18316 hybrid_clock.cc:648] HybridClock initialized: now 1790606666857421 us; error 25 us; skew 500 ppm
Sep 28 14:44:26 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606666, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:26.860105 18316 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:44:26.860918 18316 webserver.cc:460] Webserver started at http://127.1.201.62:35771/ using document root <none> and password file <none>
I20260928 14:44:26.861440 18316 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:26.861498 18316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:26.865203 18316 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:26.867300 18331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:26.868500 18316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:44:26.868634 18316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab"
format_stamp: "Formatted at 2026-09-28 14:43:44 on dist-test-slave-0n27"
server_key: "f823842a0f54d7efccec6e2385dea35a"
server_key_iv: "d0d4a4e7f74b50c73ec9b91239353c67"
server_key_version: "encryptionkey@0"
I20260928 14:44:26.869163 18316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:26.882131 18316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:26.884816 18316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:26.885006 18316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:26.893105 18316 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:34987
I20260928 14:44:26.893476 18383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:34987 every 8 connection(s)
I20260928 14:44:26.894588 18316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:26.899055 18384 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:44:26.900658 18384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Bootstrap starting.
I20260928 14:44:26.902940 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 18316
I20260928 14:44:26.903486 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:42921
--local_ip_for_outbound_sockets=127.1.201.1
--tserver_master_addrs=127.1.201.62:34987
--webserver_port=45681
--webserver_interface=127.1.201.1
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:44:26.923517 18384 log.cc:824] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:26.937963 18384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:44:26.938407 18384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Bootstrap complete.
I20260928 14:44:26.943758 18384 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:44:26.944233 18384 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca3d307a25a4e14a42e6d6d57a7b1ab, State: Initialized, Role: FOLLOWER
I20260928 14:44:26.944758 18384 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:44:26.944904 18384 raft_consensus.cc:388] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:26.944970 18384 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:26.945060 18384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:44:26.947645 18384 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:44:26.948014 18384 leader_election.cc:302] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca3d307a25a4e14a42e6d6d57a7b1ab; no voters:
I20260928 14:44:26.948370 18384 leader_election.cc:288] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:44:26.948549 18388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:44:26.948907 18388 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [term 3 LEADER]: Becoming Leader. State: Replica: bca3d307a25a4e14a42e6d6d57a7b1ab, State: Running, Role: LEADER
I20260928 14:44:26.949247 18388 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } }
I20260928 14:44:26.949615 18384 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:26.950778 18390 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca3d307a25a4e14a42e6d6d57a7b1ab. Latest consensus state: current_term: 3 leader_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } } }
I20260928 14:44:26.950892 18390 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:26.951543 18389 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca3d307a25a4e14a42e6d6d57a7b1ab" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34987 } } }
I20260928 14:44:26.951943 18389 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:26.952198 18398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:26.957129 18398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:26.957734 18398 catalog_manager.cc:1259] Loaded cluster ID: 82ec9cb8498146019336476102339e3e
I20260928 14:44:26.957823 18398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:26.960140 18398 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:26.960866 18398 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Loaded TSK: 0
I20260928 14:44:26.960942 18398 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bca3d307a25a4e14a42e6d6d57a7b1ab: Loaded TSK: 1
F20260928 14:44:26.961663 18398 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f858acf19ad google::LogMessage::Fail() at ??:0
@ 0x7f858acf488a google::LogMessage::SendToLog() at ??:0
@ 0x7f858acf14de google::LogMessage::Flush() at ??:0
@ 0x7f858acf2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f858d9b518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f858d9adf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f859565a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f859563499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8595635178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f8595676df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f859569582b std::function<>::operator()() at ??:0
@ 0x7f8595634b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8595635f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8595632579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8595676640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f858cf433d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f858cf43c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f858cf453f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f858cf34512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f858e5786db start_thread at ??:0
@ 0x7f85897ef61f clone at ??:0
W20260928 14:44:27.072239 18386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:27.072563 18386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:27.072656 18386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:27.078686 18386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:27.078758 18386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:27.078840 18386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:44:27.086374 18386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:42921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=45681
--tserver_master_addrs=127.1.201.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:27.087754 18386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:27.089233 18386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:27.101832 18432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:27.101872 18433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:27.102347 18435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:27.103400 18386 server_base.cc:1034] running on GCE node
I20260928 14:44:27.103858 18386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:27.104519 18386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:27.105690 18386 hybrid_clock.cc:648] HybridClock initialized: now 1790606667105657 us; error 48 us; skew 500 ppm
Sep 28 14:44:27 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606667, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:27.108444 18386 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:44:27.109340 18386 webserver.cc:460] Webserver started at http://127.1.201.1:45681/ using document root <none> and password file <none>
I20260928 14:44:27.109970 18386 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:27.110037 18386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:27.114010 18386 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 14:44:27.116220 18442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:27.117506 18386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:27.117637 18386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf"
format_stamp: "Formatted at 2026-09-28 14:43:44 on dist-test-slave-0n27"
server_key: "5203a68597435e3531724e9c733bcda7"
server_key_iv: "dfce6399cebc7ff523f5854686c00f8e"
server_key_version: "encryptionkey@0"
I20260928 14:44:27.118221 18386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:27.139524 18386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:27.142814 18386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:27.143021 18386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:27.143592 18386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:27.144658 18449 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:44:27.150053 18386 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:44:27.150123 18386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 14:44:27.150221 18386 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:44:27.151852 18386 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:44:27.151916 18386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 14:44:27.152192 18449 tablet_bootstrap.cc:492] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Bootstrap starting.
I20260928 14:44:27.162472 18386 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:42921
I20260928 14:44:27.162830 18556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:42921 every 8 connection(s)
I20260928 14:44:27.163456 18386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:44:27.171974 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 18386
I20260928 14:44:27.172406 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:35771
--local_ip_for_outbound_sockets=127.1.201.2
--tserver_master_addrs=127.1.201.62:34987
--webserver_port=33379
--webserver_interface=127.1.201.2
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:44:27.173312 18449 log.cc:824] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:27.180280 18449 tablet_bootstrap.cc:492] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:44:27.180636 18449 tablet_bootstrap.cc:492] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Bootstrap complete.
I20260928 14:44:27.181707 18449 ts_tablet_manager.cc:1397] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.004s
I20260928 14:44:27.184226 18449 raft_consensus.cc:348] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:27.184629 18449 raft_consensus.cc:729] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cac3ddab4ff465bb0fe08a49e6bfcaf, State: Initialized, Role: FOLLOWER
I20260928 14:44:27.185105 18449 consensus_queue.cc:260] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:27.186021 18449 ts_tablet_manager.cc:1428] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260928 14:44:27.214537 18559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:44:27.165324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:36109)
0928 14:44:27.165774 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:27.165854 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:27.166656 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:27.213591 (+ 46935us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":127,"threads_started":1}
W20260928 14:44:27.215062 18557 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:34987 (0 consecutive failures): Network error: Failed to ping master at 127.1.201.62:34987: Client connection negotiation failed: client connection to 127.1.201.62:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:44:27.215814 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:27.215518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:38841)
0928 14:44:27.215672 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:27.215749 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:44:27.216542 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:27.216324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:55267)
0928 14:44:27.216431 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:27.216482 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:44:27.216729 18557 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:44:27.290867 18560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:27.291126 18560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:27.291167 18560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:27.294950 18560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:27.294993 18560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:27.295045 18560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:44:27.300143 18560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:35771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=33379
--tserver_master_addrs=127.1.201.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:27.301103 18560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:27.302035 18560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:27.311414 18570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:27.311463 18567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:27.311513 18568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:27.311445 18560 server_base.cc:1034] running on GCE node
I20260928 14:44:27.311960 18560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:27.312508 18560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:27.313666 18560 hybrid_clock.cc:648] HybridClock initialized: now 1790606667313641 us; error 37 us; skew 500 ppm
Sep 28 14:44:27 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606667, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:27.316406 18560 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:44:27.317301 18560 webserver.cc:460] Webserver started at http://127.1.201.2:33379/ using document root <none> and password file <none>
I20260928 14:44:27.317906 18560 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:27.317970 18560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:27.321694 18560 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:27.323728 18577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:27.324927 18560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:27.325058 18560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "444a46ca7b14413da09a9027603aa7a2"
format_stamp: "Formatted at 2026-09-28 14:43:45 on dist-test-slave-0n27"
server_key: "7ae12581a3d0204e11d56108346bbf0f"
server_key_iv: "4fc5bb64675b24bca9db72d5f5f0984c"
server_key_version: "encryptionkey@0"
I20260928 14:44:27.325582 18560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:27.355249 18560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:27.358156 18560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:27.358357 18560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:27.358919 18560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:27.359716 18560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:27.359786 18560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:27.359864 18560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:27.359905 18560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:27.368758 18560 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:35771
I20260928 14:44:27.368790 18692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:35771 every 8 connection(s)
I20260928 14:44:27.369730 18560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:44:27.373442 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:27.371768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:36169)
0928 14:44:27.372216 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:27.372321 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":105,"threads_started":1}
W20260928 14:44:27.373946 18693 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:34987 (0 consecutive failures): Network error: Failed to ping master at 127.1.201.62:34987: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
I20260928 14:44:27.374471 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:27.374293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:35425)
0928 14:44:27.374384 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:27.374425 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 14:44:27.374987 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:27.374828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:39079)
0928 14:44:27.374912 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:27.374947 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 14:44:27.375149 18693 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:44:27.380014 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 18560
I20260928 14:44:27.380376 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:41945
--local_ip_for_outbound_sockets=127.1.201.3
--tserver_master_addrs=127.1.201.62:34987
--webserver_port=41459
--webserver_interface=127.1.201.3
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:27.489761 18696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:27.490012 18696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:27.490051 18696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:27.493853 18696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:27.493897 18696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:27.493950 18696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:44:27.499265 18696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:41945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=41459
--tserver_master_addrs=127.1.201.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:27.500181 18696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:27.501098 18696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:27.510218 18701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:27.510306 18696 server_base.cc:1034] running on GCE node
W20260928 14:44:27.510221 18702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:27.510241 18704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:27.510872 18696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:27.511464 18696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:27.512676 18696 hybrid_clock.cc:648] HybridClock initialized: now 1790606667512659 us; error 49 us; skew 500 ppm
Sep 28 14:44:27 dist-test-slave-0n27 krb5kdc[17232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606667, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:27.515372 18696 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:44:27.516245 18696 webserver.cc:460] Webserver started at http://127.1.201.3:41459/ using document root <none> and password file <none>
I20260928 14:44:27.516747 18696 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:27.516804 18696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:27.520431 18696 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:27.522536 18711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:27.523790 18696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:27.523907 18696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "4908a7b69cd2411ba7e69b6785aeddb7"
format_stamp: "Formatted at 2026-09-28 14:43:45 on dist-test-slave-0n27"
server_key: "76d1449e594f6b0dee29a967c5477e00"
server_key_iv: "ef3821166a470226f96c840fb7ae02d8"
server_key_version: "encryptionkey@0"
I20260928 14:44:27.524427 18696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:27.543246 18696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:27.546463 18696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:27.546679 18696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:27.547257 18696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:27.548054 18696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:27.548102 18696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:27.548169 18696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:27.548208 18696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:27.556833 18696 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:41945
I20260928 14:44:27.556869 18825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:41945 every 8 connection(s)
I20260928 14:44:27.557915 18696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:44:27.562896 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:27.560537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:39929)
0928 14:44:27.561172 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:27.561320 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":154,"threads_started":1}
W20260928 14:44:27.563381 18826 heartbeater.cc:644] Failed to heartbeat to 127.1.201.62:34987 (0 consecutive failures): Network error: Failed to ping master at 127.1.201.62:34987: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
I20260928 14:44:27.563997 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:27.563739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:44213)
0928 14:44:27.563870 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:27.563936 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 14:44:27.565575 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:27.565338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:44655)
0928 14:44:27.565471 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:27.565519 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 14:44:27.566143 18826 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:44:27.566882 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 18696
I20260928 14:44:28.217774 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:28.217351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:34181)
0928 14:44:28.217609 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:28.217665 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260928 14:44:28.376121 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:28.375722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:37897)
0928 14:44:28.375968 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:28.376023 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
I20260928 14:44:28.567026 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:28.566655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:54805)
0928 14:44:28.566881 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:28.566931 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260928 14:44:28.639037 18833 raft_consensus.cc:482] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:28.639276 18833 raft_consensus.cc:504] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:28.640511 18833 leader_election.cc:288] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771), 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945)
Sep 28 14:44:28 dist-test-slave-0n27 krb5kdc[17232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606667, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.3@KRBTEST.COM
Sep 28 14:44:28 dist-test-slave-0n27 krb5kdc[17232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606667, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.2@KRBTEST.COM
I20260928 14:44:28.657526 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:28.640745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:35771 (local address 127.1.201.1:39743)
0928 14:44:28.640887 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:28.640902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:28.641011 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:28.643193 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:28.643199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:28.643830 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:28.644711 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:28.644732 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:28.646632 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:28.646636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:28.648286 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:28.648296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:28.649216 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:28.649220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:28.649492 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:28.650312 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:28.650323 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:28.652387 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:28.654536 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:28.654540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:28.654542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:28.655019 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:28.655404 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:28.655407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:28.655409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:28.655620 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:28.656397 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:28.656404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:28.656765 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 14:44:28.656981 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 14:44:28.657331 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":530,"spinlock_wait_cycles":43776}
I20260928 14:44:28.657582 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:28.641198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:48423 (local address 127.1.201.3:41945)
0928 14:44:28.641516 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:28.641535 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:28.641569 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:44:28.642323 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:28.642330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:28.642762 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:28.643091 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:28.644894 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:28.646464 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:28.648406 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:28.649089 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:28.649255 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:28.652280 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:28.652285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:28.652296 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:28.652320 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:28.654374 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:28.655148 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:28.655154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:28.655160 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:28.655297 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:28.655738 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:28.655741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:28.655743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:28.656269 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:28.656393 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:28.657116 (+ 723us) server_negotiation.cc:299] Negotiation successful
0928 14:44:28.657399 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":121,"threads_started":1}
I20260928 14:44:28.657582 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:28.640909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:39743 (local address 127.1.201.2:35771)
0928 14:44:28.641289 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:28.641310 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:28.641370 (+ 60us) server_negotiation.cc:407] Connection header received
0928 14:44:28.642303 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:28.642313 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:28.642763 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:28.643091 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:28.644894 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:28.646464 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:28.648406 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:28.649089 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:28.649255 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:28.652570 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:28.652578 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:28.652593 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:28.652632 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:28.654448 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:28.655148 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:28.655154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:28.655171 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:28.655297 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:28.655738 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:28.655741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:28.655743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:28.656269 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:28.656393 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:28.657116 (+ 723us) server_negotiation.cc:299] Negotiation successful
0928 14:44:28.657398 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":69,"threads_started":1}
I20260928 14:44:28.657526 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:28.641012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:41945 (local address 127.1.201.1:48423)
0928 14:44:28.641289 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:28.641302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:28.641389 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:28.643193 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:28.643199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:28.643830 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:28.644711 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:28.644732 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:28.646632 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:28.646636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:28.648286 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:28.648296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:28.649216 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:28.649220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:28.649493 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:28.650233 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:28.650269 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:28.652042 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:28.654509 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:28.654517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:28.654532 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:28.655019 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:28.655404 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:28.655407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:28.655409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:28.655620 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:28.656397 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:28.656404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:28.656765 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 14:44:28.656981 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 14:44:28.657384 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":503,"spinlock_wait_cycles":44544,"thread_start_us":87,"threads_started":1}
W20260928 14:44:28.660239 18445 leader_election.cc:341] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771): Not found: Tablet not found: b47f6b1873424b68bb54ce4329ceb8d4
W20260928 14:44:28.660522 18445 leader_election.cc:341] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945): Not found: Tablet not found: b47f6b1873424b68bb54ce4329ceb8d4
I20260928 14:44:28.660583 18445 leader_election.cc:302] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cac3ddab4ff465bb0fe08a49e6bfcaf; no voters: 444a46ca7b14413da09a9027603aa7a2, 4908a7b69cd2411ba7e69b6785aeddb7
I20260928 14:44:28.660858 18833 raft_consensus.cc:2726] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:44:29.218961 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:29.218569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:46345)
0928 14:44:29.218810 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:29.218864 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260928 14:44:29.377211 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:29.376829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:49117)
0928 14:44:29.377072 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:29.377126 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260928 14:44:29.568166 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:29.567759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:35321)
0928 14:44:29.568008 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:29.568068 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260928 14:44:30.220222 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:30.219693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:36861)
0928 14:44:30.220016 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:30.220090 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":71,"threads_started":1}
I20260928 14:44:30.313331 18841 raft_consensus.cc:482] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:30.313444 18841 raft_consensus.cc:504] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:30.313906 18841 leader_election.cc:288] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771), 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945)
W20260928 14:44:30.314718 18445 leader_election.cc:341] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771): Not found: Tablet not found: b47f6b1873424b68bb54ce4329ceb8d4
W20260928 14:44:30.314874 18445 leader_election.cc:341] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945): Not found: Tablet not found: b47f6b1873424b68bb54ce4329ceb8d4
I20260928 14:44:30.314949 18445 leader_election.cc:302] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cac3ddab4ff465bb0fe08a49e6bfcaf; no voters: 444a46ca7b14413da09a9027603aa7a2, 4908a7b69cd2411ba7e69b6785aeddb7
I20260928 14:44:30.315140 18841 raft_consensus.cc:2726] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:44:30.378350 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:30.377971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:42643)
0928 14:44:30.378208 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:30.378260 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260928 14:44:30.569211 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:30.568842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:35333)
0928 14:44:30.569060 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:30.569108 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
I20260928 14:44:31.221470 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:31.221045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:57547)
0928 14:44:31.221324 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:31.221378 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
I20260928 14:44:31.379434 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:31.379050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:50239)
0928 14:44:31.379289 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:31.379344 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260928 14:44:31.570304 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:31.569915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:34379)
0928 14:44:31.570163 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:31.570216 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":88,"threads_started":1}
I20260928 14:44:32.222855 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:32.222366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:58321)
0928 14:44:32.222664 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:32.222719 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260928 14:44:32.380532 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:32.380141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:56121)
0928 14:44:32.380393 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:32.380444 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260928 14:44:32.571527 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:32.571109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:53779)
0928 14:44:32.571350 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:32.571438 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260928 14:44:32.699641 18850 raft_consensus.cc:482] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:32.699774 18850 raft_consensus.cc:504] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:32.700254 18850 leader_election.cc:288] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771), 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945)
W20260928 14:44:32.701112 18445 leader_election.cc:341] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771): Not found: Tablet not found: b47f6b1873424b68bb54ce4329ceb8d4
W20260928 14:44:32.701309 18445 leader_election.cc:341] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945): Not found: Tablet not found: b47f6b1873424b68bb54ce4329ceb8d4
I20260928 14:44:32.701380 18445 leader_election.cc:302] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cac3ddab4ff465bb0fe08a49e6bfcaf; no voters: 444a46ca7b14413da09a9027603aa7a2, 4908a7b69cd2411ba7e69b6785aeddb7
I20260928 14:44:32.701560 18850 raft_consensus.cc:2726] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:44:33.224062 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:33.223574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:45477)
0928 14:44:33.223875 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:33.223941 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260928 14:44:33.381664 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:33.381268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:43483)
0928 14:44:33.381518 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:33.381570 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260928 14:44:33.572563 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:33.572206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:36677)
0928 14:44:33.572429 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:33.572479 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260928 14:44:34.225149 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:34.224769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:42289)
0928 14:44:34.225004 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:34.225056 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260928 14:44:34.382907 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:34.382463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:47653)
0928 14:44:34.382762 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:34.382817 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260928 14:44:34.573752 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:34.573320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:46757)
0928 14:44:34.573604 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:34.573660 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":68,"threads_started":1}
I20260928 14:44:35.072666 18857 raft_consensus.cc:482] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:35.072834 18857 raft_consensus.cc:504] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:35.073282 18857 leader_election.cc:288] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771), 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945)
W20260928 14:44:35.074187 18445 leader_election.cc:341] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771): Not found: Tablet not found: b47f6b1873424b68bb54ce4329ceb8d4
W20260928 14:44:35.074380 18445 leader_election.cc:341] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945): Not found: Tablet not found: b47f6b1873424b68bb54ce4329ceb8d4
I20260928 14:44:35.074440 18445 leader_election.cc:302] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cac3ddab4ff465bb0fe08a49e6bfcaf; no voters: 444a46ca7b14413da09a9027603aa7a2, 4908a7b69cd2411ba7e69b6785aeddb7
I20260928 14:44:35.074569 18857 raft_consensus.cc:2726] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:44:35.226466 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:35.226042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:60891)
0928 14:44:35.226301 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:35.226367 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260928 14:44:35.384091 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:35.383683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:34057)
0928 14:44:35.383937 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:35.383999 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260928 14:44:35.575241 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:35.574811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:38435)
0928 14:44:35.575089 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:35.575149 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20260928 14:44:36.227658 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:36.227225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:60331)
0928 14:44:36.227499 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:36.227554 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260928 14:44:36.385291 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:36.384873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:56773)
0928 14:44:36.385143 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:36.385200 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260928 14:44:36.576419 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:36.576016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:37779)
0928 14:44:36.576270 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:36.576327 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260928 14:44:37.228966 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:37.228526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:55445)
0928 14:44:37.228782 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:37.228834 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
I20260928 14:44:37.386510 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:37.386117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:36915)
0928 14:44:37.386351 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:37.386400 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260928 14:44:37.577540 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:37.577150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:54507)
0928 14:44:37.577392 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:37.577448 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
I20260928 14:44:38.230186 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:38.229730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:58041)
0928 14:44:38.230038 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:38.230092 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":123,"threads_started":1}
I20260928 14:44:38.387584 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:38.387198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:37755)
0928 14:44:38.387441 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:38.387490 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260928 14:44:38.578734 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:38.578319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:34151)
0928 14:44:38.578580 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:38.578636 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260928 14:44:39.231375 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:39.230982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:60975)
0928 14:44:39.231223 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:39.231280 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260928 14:44:39.388727 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:39.388284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:57733)
0928 14:44:39.388574 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:39.388631 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260928 14:44:39.579774 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:39.579395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:50061)
0928 14:44:39.579623 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:39.579682 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260928 14:44:40.232486 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:40.232083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:47485)
0928 14:44:40.232340 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:40.232392 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
I20260928 14:44:40.389912 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:40.389472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:47771)
0928 14:44:40.389712 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:40.389763 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
I20260928 14:44:40.580895 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:40.580473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:54809)
0928 14:44:40.580748 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:40.580801 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260928 14:44:40.795656 18876 raft_consensus.cc:482] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:40.795789 18876 raft_consensus.cc:504] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "444a46ca7b14413da09a9027603aa7a2" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 35771 } } peers { permanent_uuid: "4908a7b69cd2411ba7e69b6785aeddb7" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41945 } } peers { permanent_uuid: "4cac3ddab4ff465bb0fe08a49e6bfcaf" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42921 } }
I20260928 14:44:40.796239 18876 leader_election.cc:288] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771), 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945)
W20260928 14:44:40.797012 18445 leader_election.cc:341] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 444a46ca7b14413da09a9027603aa7a2 (127.1.201.2:35771): Not found: Tablet not found: b47f6b1873424b68bb54ce4329ceb8d4
W20260928 14:44:40.797143 18445 leader_election.cc:341] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4908a7b69cd2411ba7e69b6785aeddb7 (127.1.201.3:41945): Not found: Tablet not found: b47f6b1873424b68bb54ce4329ceb8d4
I20260928 14:44:40.797188 18445 leader_election.cc:302] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cac3ddab4ff465bb0fe08a49e6bfcaf; no voters: 444a46ca7b14413da09a9027603aa7a2, 4908a7b69cd2411ba7e69b6785aeddb7
I20260928 14:44:40.797317 18876 raft_consensus.cc:2726] T b47f6b1873424b68bb54ce4329ceb8d4 P 4cac3ddab4ff465bb0fe08a49e6bfcaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:44:41.233623 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:41.233213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:40569)
0928 14:44:41.233472 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:41.233527 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260928 14:44:41.391023 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:41.390614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:36927)
0928 14:44:41.390874 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:41.390929 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
I20260928 14:44:41.582155 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:41.581685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:52943)
0928 14:44:41.582007 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:41.582059 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":162,"threads_started":1}
I20260928 14:44:42.234987 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:42.234561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.1:35949)
0928 14:44:42.234840 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:42.234894 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
I20260928 14:44:42.392280 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:42.391848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.2:51593)
0928 14:44:42.392120 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:42.392189 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
I20260928 14:44:42.575407 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 18386
I20260928 14:44:42.581382 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 18560
I20260928 14:44:42.583465 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:42.583070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34987 (local address 127.1.201.3:34965)
0928 14:44:42.583328 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:42.583378 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.62:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260928 14:44:42.586938 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 18696
2026-09-28T14:44:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59294 ms)
[----------] 30 tests from SecurityITest (237735 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:44:42 dist-test-slave-0n27 krb5kdc[18885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:44:42 dist-test-slave-0n27 krb5kdc[18885](info): set up 2 sockets
Sep 28 14:44:42 dist-test-slave-0n27 krb5kdc[18885](info): commencing operation
krb5kdc: starting...
W20260928 14:44:43.626428 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:44:43 dist-test-slave-0n27 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:44:43.744721 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:34249
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:34249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:43.853396 18899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:43.853631 18899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:43.853669 18899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:43.857580 18899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:43.857632 18899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:43.857651 18899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:43.857667 18899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:43.857682 18899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:43.862766 18899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:34249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:34249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:43.863654 18899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:43.864565 18899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:43.873107 18905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:43.873149 18904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:43.873200 18899 server_base.cc:1034] running on GCE node
W20260928 14:44:43.873132 18907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:43.873909 18899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:43.875082 18899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:43.876272 18899 hybrid_clock.cc:648] HybridClock initialized: now 1790606683876258 us; error 25 us; skew 500 ppm
Sep 28 14:44:43 dist-test-slave-0n27 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606683, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:43.879060 18899 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:44:43.879946 18899 webserver.cc:460] Webserver started at http://127.1.201.62:39959/ using document root <none> and password file <none>
I20260928 14:44:43.880492 18899 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:43.880581 18899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:43.880916 18899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:43.882947 18899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "5eb7e563d0854be0bf39f6a45b174e13"
format_stamp: "Formatted at 2026-09-28 14:44:43 on dist-test-slave-0n27"
server_key: "f7f3f0e7b3fb2a2074c9eb6e6e9006e1"
server_key_iv: "bb865018c7aa72bb6509dce76a339efc"
server_key_version: "encryptionkey@0"
I20260928 14:44:43.883540 18899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "5eb7e563d0854be0bf39f6a45b174e13"
format_stamp: "Formatted at 2026-09-28 14:44:43 on dist-test-slave-0n27"
server_key: "f7f3f0e7b3fb2a2074c9eb6e6e9006e1"
server_key_iv: "bb865018c7aa72bb6509dce76a339efc"
server_key_version: "encryptionkey@0"
I20260928 14:44:43.887109 18899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:44:43.889380 18914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:43.890581 18899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:43.890723 18899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "5eb7e563d0854be0bf39f6a45b174e13"
format_stamp: "Formatted at 2026-09-28 14:44:43 on dist-test-slave-0n27"
server_key: "f7f3f0e7b3fb2a2074c9eb6e6e9006e1"
server_key_iv: "bb865018c7aa72bb6509dce76a339efc"
server_key_version: "encryptionkey@0"
I20260928 14:44:43.890833 18899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:43.915369 18899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:43.918221 18899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:43.918426 18899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:43.925419 18899 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:34249
I20260928 14:44:43.925439 18966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:34249 every 8 connection(s)
I20260928 14:44:43.926330 18899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:43.928957 18967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:43.930584 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 18899
I20260928 14:44:43.930730 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:43.931016 1828 external_mini_cluster.cc:1428] Setting key ddd9dacd99d1000a5ee3c14444ba2ccb
I20260928 14:44:43.935662 18967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13: Bootstrap starting.
I20260928 14:44:43.938215 18967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:43.938987 18967 log.cc:824] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:44:43 dist-test-slave-0n27 krb5kdc[18885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:44:43.941143 18967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13: No bootstrap required, opened a new log
I20260928 14:44:43.945890 18967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb7e563d0854be0bf39f6a45b174e13" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34249 } }
I20260928 14:44:43.946146 18967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:43.946161 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:43.932425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.1.201.62:34249)
0928 14:44:43.932873 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:43.932892 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:43.932933 (+ 41us) server_negotiation.cc:407] Connection header received
0928 14:44:43.933773 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:43.933817 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:43.934240 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:43.934603 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:43.935213 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:43.936666 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:43.937521 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:43.938231 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:43.938398 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:43.941269 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:43.941275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:43.941291 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:43.941319 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:43.943209 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:43.943641 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:43.943649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:43.943657 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:43.943804 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:43.944059 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:43.944062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:43.944064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:43.944541 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:43.944740 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:43.944919 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 14:44:43.945221 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":101,"threads_started":1}
I20260928 14:44:43.946221 18967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb7e563d0854be0bf39f6a45b174e13, State: Initialized, Role: FOLLOWER
I20260928 14:44:43.946686 18967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb7e563d0854be0bf39f6a45b174e13" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34249 } }
I20260928 14:44:43.946844 18967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:43.946933 18967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:43.947049 18967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:43.948139 18967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb7e563d0854be0bf39f6a45b174e13" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34249 } }
I20260928 14:44:43.948488 18967 leader_election.cc:302] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eb7e563d0854be0bf39f6a45b174e13; no voters:
I20260928 14:44:43.948827 18967 leader_election.cc:288] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:43.948933 18972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:43.949164 18972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [term 1 LEADER]: Becoming Leader. State: Replica: 5eb7e563d0854be0bf39f6a45b174e13, State: Running, Role: LEADER
I20260928 14:44:43.949499 18972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb7e563d0854be0bf39f6a45b174e13" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34249 } }
I20260928 14:44:43.950074 18967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:43.951324 18974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eb7e563d0854be0bf39f6a45b174e13. Latest consensus state: current_term: 1 leader_uuid: "5eb7e563d0854be0bf39f6a45b174e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb7e563d0854be0bf39f6a45b174e13" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34249 } } }
I20260928 14:44:43.951329 18973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eb7e563d0854be0bf39f6a45b174e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb7e563d0854be0bf39f6a45b174e13" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34249 } } }
I20260928 14:44:43.951474 18974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:43.951475 18973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:43.952383 18981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:43.955297 18981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:43.960145 18981 catalog_manager.cc:1347] Generated new cluster ID: 0cefabae9aaa4f4887ae0d6821da9cb7
I20260928 14:44:43.960206 18981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:43.979611 18981 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:43.980842 18981 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:43.985824 18981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13: Generated new TSK 0
I20260928 14:44:43.986586 18981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:44:44.022047 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34249
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:44.129732 18993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:44.129997 18993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:44.130038 18993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:44.133770 18993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:44.133872 18993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:44.133956 18993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:44:44.139117 18993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:44.140107 18993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:44.141080 18993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:44.150144 19001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:44.150236 18993 server_base.cc:1034] running on GCE node
W20260928 14:44:44.150180 18999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:44.150144 18998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:44.150866 18993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:44.151436 18993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:44.152663 18993 hybrid_clock.cc:648] HybridClock initialized: now 1790606684152654 us; error 30 us; skew 500 ppm
Sep 28 14:44:44 dist-test-slave-0n27 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606684, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:44.155345 18993 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:44:44.156152 18993 webserver.cc:460] Webserver started at http://127.1.201.1:42779/ using document root <none> and password file <none>
I20260928 14:44:44.156702 18993 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:44.156790 18993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:44.157016 18993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:44.159037 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "36d2ab7402e64e238a26e97181cb56e7"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "7b934a5a2e43dfe95b5a2d0ebd603aa2"
server_key_iv: "d235116c2ed546c323597d88387b8728"
server_key_version: "encryptionkey@0"
I20260928 14:44:44.159646 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "36d2ab7402e64e238a26e97181cb56e7"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "7b934a5a2e43dfe95b5a2d0ebd603aa2"
server_key_iv: "d235116c2ed546c323597d88387b8728"
server_key_version: "encryptionkey@0"
I20260928 14:44:44.164057 18993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:44:44.166424 19008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:44.167553 18993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:44.167687 18993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "36d2ab7402e64e238a26e97181cb56e7"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "7b934a5a2e43dfe95b5a2d0ebd603aa2"
server_key_iv: "d235116c2ed546c323597d88387b8728"
server_key_version: "encryptionkey@0"
I20260928 14:44:44.167796 18993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:44.180506 18993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:44.183378 18993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:44.183578 18993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:44.184120 18993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:44.184899 18993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:44.184967 18993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:44.185031 18993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:44.185081 18993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:44.193746 18993 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:36417
I20260928 14:44:44.193797 19122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:36417 every 8 connection(s)
I20260928 14:44:44.194669 18993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:44:44.197676 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 18993
I20260928 14:44:44.197851 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:44:44.198252 1828 external_mini_cluster.cc:1428] Setting key 51b960700469f5c371700724974a1088
Sep 28 14:44:44 dist-test-slave-0n27 krb5kdc[18885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606684, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
I20260928 14:44:44.209542 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.196409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:45549 (local address 127.1.201.62:34249)
0928 14:44:44.196568 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:44.196574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:44.197370 (+ 796us) server_negotiation.cc:407] Connection header received
0928 14:44:44.198970 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:44.198974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:44.199027 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:44.199124 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:44.200382 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.201245 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.202447 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.202863 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.202986 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.205712 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:44.205717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:44.205725 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:44.205749 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:44.207284 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:44.207811 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:44.207816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:44.207818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:44.207899 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:44.208335 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:44.208338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:44.208339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:44.208513 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:44.208595 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:44.209256 (+ 661us) server_negotiation.cc:299] Negotiation successful
0928 14:44:44.209416 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:44:44.210209 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.196805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34249 (local address 127.1.201.1:45549)
0928 14:44:44.197227 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:44.197268 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:44.198774 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:44.199347 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:44.199356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:44.199778 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:44.200236 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.200249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.201371 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.201374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.202326 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.202335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.203036 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.203043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.203271 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.203870 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:44.203895 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:44.205535 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:44.207417 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:44.207423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:44.207437 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:44.207707 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:44.208032 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:44.208035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:44.208037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:44.208237 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:44.208615 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:44.208620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:44.208857 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 14:44:44.209013 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 14:44:44.209312 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
I20260928 14:44:44.211448 19123 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34249
I20260928 14:44:44.211699 19123 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:44.212289 19123 heartbeater.cc:511] Master 127.1.201.62:34249 requested a full tablet report, sending...
I20260928 14:44:44.214111 18931 ts_manager.cc:194] Registered new tserver with Master: 36d2ab7402e64e238a26e97181cb56e7 (127.1.201.1:36417)
I20260928 14:44:44.215268 18931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:45549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:44:44.226804 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34249
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:44.334484 19128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:44.334750 19128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:44.334831 19128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:44.338850 19128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:44.338902 19128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:44.339016 19128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:44:44.344165 19128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:44.345224 19128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:44.346451 19128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:44.355880 19133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:44.355998 19128 server_base.cc:1034] running on GCE node
W20260928 14:44:44.355919 19136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:44.356132 19134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:44.356535 19128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:44.357095 19128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:44.358245 19128 hybrid_clock.cc:648] HybridClock initialized: now 1790606684358231 us; error 26 us; skew 500 ppm
Sep 28 14:44:44 dist-test-slave-0n27 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606684, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:44.360925 19128 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:44:44.361727 19128 webserver.cc:460] Webserver started at http://127.1.201.2:39411/ using document root <none> and password file <none>
I20260928 14:44:44.362283 19128 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:44.362371 19128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:44.362588 19128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:44.364477 19128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "455a4c2bf6d545feb792f749c645f80d"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "c57df08e9435dad8da4610ef4d35ab6a"
server_key_iv: "048516adbb03269a11c3fc96b3b45626"
server_key_version: "encryptionkey@0"
I20260928 14:44:44.365012 19128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "455a4c2bf6d545feb792f749c645f80d"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "c57df08e9435dad8da4610ef4d35ab6a"
server_key_iv: "048516adbb03269a11c3fc96b3b45626"
server_key_version: "encryptionkey@0"
I20260928 14:44:44.368501 19128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 14:44:44.370849 19143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:44.372035 19128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:44:44.372161 19128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "455a4c2bf6d545feb792f749c645f80d"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "c57df08e9435dad8da4610ef4d35ab6a"
server_key_iv: "048516adbb03269a11c3fc96b3b45626"
server_key_version: "encryptionkey@0"
I20260928 14:44:44.372265 19128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:44.397745 19128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:44.400470 19128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:44.400687 19128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:44.401244 19128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:44.402204 19128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:44.402251 19128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:44.402290 19128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:44.402349 19128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:44.410977 19128 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:36447
I20260928 14:44:44.411000 19257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:36447 every 8 connection(s)
I20260928 14:44:44.411805 19128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:44:44.412670 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 19128
I20260928 14:44:44.412813 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:44:44.413122 1828 external_mini_cluster.cc:1428] Setting key ef57daa4be1ff0f2f06c3ac5671f8140
Sep 28 14:44:44 dist-test-slave-0n27 krb5kdc[18885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606684, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
I20260928 14:44:44.427047 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.413651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:48747 (local address 127.1.201.62:34249)
0928 14:44:44.413819 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:44.413823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:44.414814 (+ 991us) server_negotiation.cc:407] Connection header received
0928 14:44:44.416718 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:44.416724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:44.416785 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:44.416878 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:44.418013 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.418568 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.420412 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.420697 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.420777 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.423501 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:44.423510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:44.423512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:44.423535 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:44.424910 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:44.425383 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:44.425386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:44.425388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:44.425463 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:44.425892 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:44.425895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:44.425897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:44.426111 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:44.426255 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:44.426834 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 14:44:44.426930 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:44:44.428187 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.414116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34249 (local address 127.1.201.2:48747)
0928 14:44:44.414671 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:44.414728 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:44.416472 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:44.417005 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:44.417011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:44.417351 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:44.417883 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.417899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.418707 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.418713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.420265 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.420281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.420867 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.420873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.421120 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.421724 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:44.421745 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:44.423341 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:44.425017 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:44.425023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:44.425036 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:44.425280 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:44.425600 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:44.425603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:44.425605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:44.425768 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:44.426255 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:44.426262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:44.426541 (+ 279us) client_negotiation.cc:769] Sending connection context
0928 14:44:44.426766 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 14:44:44.427111 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":146,"threads_started":1}
I20260928 14:44:44.429335 19258 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34249
I20260928 14:44:44.429582 19258 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:44.430102 19258 heartbeater.cc:511] Master 127.1.201.62:34249 requested a full tablet report, sending...
I20260928 14:44:44.431252 18931 ts_manager.cc:194] Registered new tserver with Master: 455a4c2bf6d545feb792f749c645f80d (127.1.201.2:36447)
I20260928 14:44:44.432102 18931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:48747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:44:44.441490 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34249
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:44.550359 19263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:44.550688 19263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:44.550736 19263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:44.554963 19263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:44.555012 19263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:44.555085 19263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:44:44.561044 19263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:44.562397 19263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:44.563799 19263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:44.576336 19268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:44.576508 19271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:44.576759 19263 server_base.cc:1034] running on GCE node
W20260928 14:44:44.576829 19269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:44.577320 19263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:44.578142 19263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:44.579329 19263 hybrid_clock.cc:648] HybridClock initialized: now 1790606684579293 us; error 52 us; skew 500 ppm
Sep 28 14:44:44 dist-test-slave-0n27 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606684, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:44.582506 19263 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:44:44.583627 19263 webserver.cc:460] Webserver started at http://127.1.201.3:37007/ using document root <none> and password file <none>
I20260928 14:44:44.584245 19263 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:44.584344 19263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:44.584611 19263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:44.586963 19263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "b66bfa111c164faca8c284f752f5ff1f"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "ef9e74453619bbe34f27f51293e4fcdd"
server_key_iv: "51f1025244eb49b7296f3be22c99f812"
server_key_version: "encryptionkey@0"
I20260928 14:44:44.587447 19263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "b66bfa111c164faca8c284f752f5ff1f"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "ef9e74453619bbe34f27f51293e4fcdd"
server_key_iv: "51f1025244eb49b7296f3be22c99f812"
server_key_version: "encryptionkey@0"
I20260928 14:44:44.590972 19263 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:44:44.593194 19278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:44.594563 19263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:44.594698 19263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "b66bfa111c164faca8c284f752f5ff1f"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "ef9e74453619bbe34f27f51293e4fcdd"
server_key_iv: "51f1025244eb49b7296f3be22c99f812"
server_key_version: "encryptionkey@0"
I20260928 14:44:44.594822 19263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:44.610234 19263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:44.613358 19263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:44.613554 19263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:44.614217 19263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:44.615031 19263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:44.615085 19263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:44.615127 19263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:44.615181 19263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:44.623631 19263 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:34339
I20260928 14:44:44.623653 19391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:34339 every 8 connection(s)
I20260928 14:44:44.624506 19263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:44:44.627367 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 19263
I20260928 14:44:44.627483 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:44:44.627862 1828 external_mini_cluster.cc:1428] Setting key c5b45e6f1c3391c9650ddf38b9ced6f7
Sep 28 14:44:44 dist-test-slave-0n27 krb5kdc[18885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606684, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:44:44.637398 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.626305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:43191 (local address 127.1.201.62:34249)
0928 14:44:44.626454 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:44.626458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:44.627265 (+ 807us) server_negotiation.cc:407] Connection header received
0928 14:44:44.628903 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:44.628905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:44.628948 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:44.629038 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:44.630323 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.630832 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.631904 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.632110 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.632199 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.634582 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:44.634585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:44.634588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:44.634609 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:44.635785 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:44.636218 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:44.636220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:44.636221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:44.636286 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:44.636552 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:44.636554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:44.636555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:44.636722 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:44.636794 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:44.637200 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 14:44:44.637286 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:44:44.638239 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.626638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34249 (local address 127.1.201.3:43191)
0928 14:44:44.627054 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:44.627134 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:44.628671 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:44.629213 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:44.629222 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:44.629582 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:44.630198 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.630213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.630934 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.630936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.631796 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.631805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.632244 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.632247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.632380 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.632993 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:44.633020 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:44.634441 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:44.635886 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:44.635890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:44.635900 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:44.636138 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:44.636357 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:44.636359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:44.636360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:44.636477 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:44.636803 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:44.636807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:44.637009 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 14:44:44.637153 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 14:44:44.637377 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":95,"threads_started":1}
I20260928 14:44:44.639305 19392 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34249
I20260928 14:44:44.639539 19392 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:44.640000 19392 heartbeater.cc:511] Master 127.1.201.62:34249 requested a full tablet report, sending...
I20260928 14:44:44.641085 18931 ts_manager.cc:194] Registered new tserver with Master: b66bfa111c164faca8c284f752f5ff1f (127.1.201.3:34339)
I20260928 14:44:44.641681 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:44:44.641815 18931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:43191
I20260928 14:44:44.650954 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.643724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.1.201.62:34249)
0928 14:44:44.643856 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:44.643861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:44.644037 (+ 176us) server_negotiation.cc:407] Connection header received
0928 14:44:44.644196 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:44.644199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:44.644241 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:44.644334 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:44.644995 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.645488 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.646413 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.646629 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.646733 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.648049 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:44.648053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:44.648056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:44.648090 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:44.649279 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:44.649694 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:44.649697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:44.649699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:44.649774 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:44.650078 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:44.650081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:44.650089 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:44.650303 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:44.650437 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:44.650679 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 14:44:44.650783 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:44:44.653700 18931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:44:44.656610 18931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:44:44.670722 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.664478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:36447 (local address 127.0.0.1:36138)
0928 14:44:44.664931 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:44.664974 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:44.665103 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:44.666741 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:44.666744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:44.666761 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:44.666856 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.666862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.667828 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.667832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.669112 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.669120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.670233 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.670236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.670332 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.670366 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 14:44:44.670462 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 14:44:44.670576 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":72,"spinlock_wait_cycles":7936,"thread_start_us":74,"threads_started":1}
I20260928 14:44:44.671175 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.665236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.1.201.1:36417)
0928 14:44:44.665625 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:44.665637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:44.665661 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:44:44.665741 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:44.665746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:44.665993 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:44.666159 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:44.666447 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.667635 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.669397 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.670130 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.670219 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.670355 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:44.670919 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 14:44:44.671031 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":123,"threads_started":1}
I20260928 14:44:44.671319 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.664576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:36417 (local address 127.0.0.1:37302)
0928 14:44:44.664932 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:44.664974 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:44.665108 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:44.666163 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:44.666178 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:44.666200 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:44.666344 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.666352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.667774 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.667779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.669287 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.669293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.670409 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.670413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.670517 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.670535 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:44:44.670609 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:44:44.670664 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":140,"spinlock_wait_cycles":3328,"thread_start_us":109,"threads_started":1}
I20260928 14:44:44.671397 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.665439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.1.201.3:34339)
0928 14:44:44.665841 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:44.665851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:44.665870 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:44:44.666111 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:44.666116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:44.666291 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:44.666430 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:44.666811 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.667864 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.669880 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.670517 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.670644 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.671008 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:44.671100 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 14:44:44.671197 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":92,"threads_started":1}
I20260928 14:44:44.671446 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.665153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:34339 (local address 127.0.0.1:52182)
0928 14:44:44.665409 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:44.665422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:44.665506 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:44.666578 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:44.666580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:44.666599 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:44.666681 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.666687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.667989 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.667994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.669281 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.669291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.670675 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.670679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.670757 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.670779 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:44:44.671040 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 14:44:44.671151 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":32,"thread_start_us":91,"threads_started":1}
I20260928 14:44:44.671804 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.664585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.1.201.2:36447)
0928 14:44:44.665945 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:44.666345 (+ 400us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:44.666374 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:44:44.666453 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:44.666458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:44.666611 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:44.666767 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:44.666962 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.667689 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.669231 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.670075 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.671067 (+ 992us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.671226 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:44.671283 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 14:44:44.671382 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":94,"threads_started":1}
I20260928 14:44:44.674232 19326 tablet_service.cc:1467] Processing CreateTablet for tablet f53e8e60aa39496a9ea4c5cef44b249d (DEFAULT_TABLE table=test-table [id=10558ab09d0e412b8ce6d5e468914a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:44.674369 19192 tablet_service.cc:1467] Processing CreateTablet for tablet f53e8e60aa39496a9ea4c5cef44b249d (DEFAULT_TABLE table=test-table [id=10558ab09d0e412b8ce6d5e468914a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:44.674613 19057 tablet_service.cc:1467] Processing CreateTablet for tablet f53e8e60aa39496a9ea4c5cef44b249d (DEFAULT_TABLE table=test-table [id=10558ab09d0e412b8ce6d5e468914a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:44.675298 19326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53e8e60aa39496a9ea4c5cef44b249d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:44.675405 19192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53e8e60aa39496a9ea4c5cef44b249d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:44.675405 19057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53e8e60aa39496a9ea4c5cef44b249d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:44.681504 19409 tablet_bootstrap.cc:492] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: Bootstrap starting.
I20260928 14:44:44.682807 19410 tablet_bootstrap.cc:492] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7: Bootstrap starting.
I20260928 14:44:44.683189 19411 tablet_bootstrap.cc:492] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: Bootstrap starting.
I20260928 14:44:44.684109 19409 tablet_bootstrap.cc:654] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:44.684844 19410 tablet_bootstrap.cc:654] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:44.685209 19411 tablet_bootstrap.cc:654] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:44.685400 19409 log.cc:824] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:44.686025 19410 log.cc:824] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:44.686064 19411 log.cc:824] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:44.687721 19410 tablet_bootstrap.cc:492] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7: No bootstrap required, opened a new log
I20260928 14:44:44.687881 19411 tablet_bootstrap.cc:492] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: No bootstrap required, opened a new log
I20260928 14:44:44.687886 19409 tablet_bootstrap.cc:492] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: No bootstrap required, opened a new log
I20260928 14:44:44.687943 19410 ts_tablet_manager.cc:1397] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 14:44:44.688060 19411 ts_tablet_manager.cc:1397] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:44:44.688060 19409 ts_tablet_manager.cc:1397] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 14:44:44.690742 19410 raft_consensus.cc:348] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455a4c2bf6d545feb792f749c645f80d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36447 } } peers { permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 } } peers { permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 } }
I20260928 14:44:44.690961 19410 raft_consensus.cc:374] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:44.691010 19410 raft_consensus.cc:729] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d2ab7402e64e238a26e97181cb56e7, State: Initialized, Role: FOLLOWER
I20260928 14:44:44.691431 19410 consensus_queue.cc:260] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455a4c2bf6d545feb792f749c645f80d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36447 } } peers { permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 } } peers { permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 } }
I20260928 14:44:44.691766 19409 raft_consensus.cc:348] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455a4c2bf6d545feb792f749c645f80d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36447 } } peers { permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 } } peers { permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 } }
I20260928 14:44:44.691766 19411 raft_consensus.cc:348] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455a4c2bf6d545feb792f749c645f80d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36447 } } peers { permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 } } peers { permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 } }
I20260928 14:44:44.692009 19409 raft_consensus.cc:374] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:44.692009 19411 raft_consensus.cc:374] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:44.692028 19123 heartbeater.cc:503] Master 127.1.201.62:34249 was elected leader, sending a full tablet report...
I20260928 14:44:44.692058 19411 raft_consensus.cc:729] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455a4c2bf6d545feb792f749c645f80d, State: Initialized, Role: FOLLOWER
I20260928 14:44:44.692058 19409 raft_consensus.cc:729] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66bfa111c164faca8c284f752f5ff1f, State: Initialized, Role: FOLLOWER
I20260928 14:44:44.692425 19410 ts_tablet_manager.cc:1428] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20260928 14:44:44.692431 19409 consensus_queue.cc:260] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455a4c2bf6d545feb792f749c645f80d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36447 } } peers { permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 } } peers { permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 } }
I20260928 14:44:44.692693 19411 consensus_queue.cc:260] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455a4c2bf6d545feb792f749c645f80d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36447 } } peers { permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 } } peers { permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 } }
I20260928 14:44:44.693338 19409 ts_tablet_manager.cc:1428] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:44:44.693449 19411 ts_tablet_manager.cc:1428] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:44:44.693688 19392 heartbeater.cc:503] Master 127.1.201.62:34249 was elected leader, sending a full tablet report...
I20260928 14:44:44.693820 19258 heartbeater.cc:503] Master 127.1.201.62:34249 was elected leader, sending a full tablet report...
W20260928 14:44:44.695849 19124 tablet.cc:2369] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:44.754734 19417 raft_consensus.cc:482] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:44.754972 19417 raft_consensus.cc:504] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455a4c2bf6d545feb792f749c645f80d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36447 } } peers { permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 } } peers { permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 } }
I20260928 14:44:44.756193 19417 leader_election.cc:288] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d2ab7402e64e238a26e97181cb56e7 (127.1.201.1:36417), b66bfa111c164faca8c284f752f5ff1f (127.1.201.3:34339)
I20260928 14:44:44.760301 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.756530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:35549 (local address 127.1.201.3:34339)
0928 14:44:44.756659 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:44.756662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:44.756755 (+ 93us) server_negotiation.cc:407] Connection header received
0928 14:44:44.756976 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:44.756982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:44.757047 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:44.757136 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:44.757446 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.758124 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.759342 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.759732 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.759850 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.759908 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:44.760105 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 14:44:44.760174 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:44:44.760304 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.756442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:34339 (local address 127.1.201.2:35549)
0928 14:44:44.756660 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:44.756675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:44.756786 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:44.757159 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:44.757162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:44.757190 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:44.757326 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.757334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.758243 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.758247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.759217 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.759227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.759861 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.759865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.759970 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.759996 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:44:44.760084 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:44:44.760155 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 14:44:44.760315 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.756530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:58489 (local address 127.1.201.1:36417)
0928 14:44:44.756655 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:44.756660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:44.757049 (+ 389us) server_negotiation.cc:407] Connection header received
0928 14:44:44.757166 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:44.757169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:44.757216 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:44.757295 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:44.757585 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.758237 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.759397 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.759765 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.759876 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.759937 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:44.760118 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 14:44:44.760192 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:44:44.760356 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.756442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:36417 (local address 127.1.201.2:58489)
0928 14:44:44.756941 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:44.756954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:44.757065 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:44.757321 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:44.757323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:44.757337 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:44.757471 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.757477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.758363 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.758366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.759278 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:44.759287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.759881 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.759885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:44.759991 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.760012 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:44:44.760102 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 14:44:44.760179 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":31,"thread_start_us":130,"threads_started":1}
I20260928 14:44:44.760973 19077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53e8e60aa39496a9ea4c5cef44b249d" candidate_uuid: "455a4c2bf6d545feb792f749c645f80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d2ab7402e64e238a26e97181cb56e7" is_pre_election: true
I20260928 14:44:44.761106 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53e8e60aa39496a9ea4c5cef44b249d" candidate_uuid: "455a4c2bf6d545feb792f749c645f80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66bfa111c164faca8c284f752f5ff1f" is_pre_election: true
I20260928 14:44:44.761298 19077 raft_consensus.cc:2454] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 455a4c2bf6d545feb792f749c645f80d in term 0.
I20260928 14:44:44.761327 19346 raft_consensus.cc:2454] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 455a4c2bf6d545feb792f749c645f80d in term 0.
I20260928 14:44:44.761845 19146 leader_election.cc:302] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36d2ab7402e64e238a26e97181cb56e7, 455a4c2bf6d545feb792f749c645f80d, b66bfa111c164faca8c284f752f5ff1f; no voters:
I20260928 14:44:44.762159 19417 raft_consensus.cc:2781] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:44:44.762264 19417 raft_consensus.cc:482] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:44:44.762323 19417 raft_consensus.cc:3037] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:44.763336 19417 raft_consensus.cc:504] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455a4c2bf6d545feb792f749c645f80d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36447 } } peers { permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 } } peers { permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 } }
I20260928 14:44:44.763674 19417 leader_election.cc:288] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [CANDIDATE]: Term 1 election: Requested vote from peers 36d2ab7402e64e238a26e97181cb56e7 (127.1.201.1:36417), b66bfa111c164faca8c284f752f5ff1f (127.1.201.3:34339)
I20260928 14:44:44.764034 19077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53e8e60aa39496a9ea4c5cef44b249d" candidate_uuid: "455a4c2bf6d545feb792f749c645f80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d2ab7402e64e238a26e97181cb56e7"
I20260928 14:44:44.764034 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53e8e60aa39496a9ea4c5cef44b249d" candidate_uuid: "455a4c2bf6d545feb792f749c645f80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66bfa111c164faca8c284f752f5ff1f"
I20260928 14:44:44.764154 19077 raft_consensus.cc:3037] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:44.764154 19346 raft_consensus.cc:3037] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:44.765383 19077 raft_consensus.cc:2454] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 455a4c2bf6d545feb792f749c645f80d in term 1.
I20260928 14:44:44.765383 19346 raft_consensus.cc:2454] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 455a4c2bf6d545feb792f749c645f80d in term 1.
I20260928 14:44:44.765694 19144 leader_election.cc:302] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36d2ab7402e64e238a26e97181cb56e7, 455a4c2bf6d545feb792f749c645f80d, b66bfa111c164faca8c284f752f5ff1f; no voters:
I20260928 14:44:44.765889 19417 raft_consensus.cc:2781] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:44.766130 19417 raft_consensus.cc:686] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 1 LEADER]: Becoming Leader. State: Replica: 455a4c2bf6d545feb792f749c645f80d, State: Running, Role: LEADER
I20260928 14:44:44.766484 19417 consensus_queue.cc:237] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455a4c2bf6d545feb792f749c645f80d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36447 } } peers { permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 } } peers { permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 } }
I20260928 14:44:44.769562 18930 catalog_manager.cc:5495] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d reported cstate change: term changed from 0 to 1, leader changed from <none> to 455a4c2bf6d545feb792f749c645f80d (127.1.201.2). New cstate: current_term: 1 leader_uuid: "455a4c2bf6d545feb792f749c645f80d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455a4c2bf6d545feb792f749c645f80d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 36447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20260928 14:44:44.789820 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.786654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.1.201.2:36447)
0928 14:44:44.786772 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:44.786778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:44.786794 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:44:44.786915 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:44.786917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:44.786966 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:44.787045 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:44:44.787249 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.787742 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.788588 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:44.788788 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:44.788869 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:44.789108 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:44:44.789483 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:44:44.789566 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:44.789618 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 14:44:44.789660 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:44:44.795295 19346 raft_consensus.cc:1261] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f [term 1 FOLLOWER]: Refusing update from remote peer 455a4c2bf6d545feb792f749c645f80d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:44.795316 19077 raft_consensus.cc:1261] T f53e8e60aa39496a9ea4c5cef44b249d P 36d2ab7402e64e238a26e97181cb56e7 [term 1 FOLLOWER]: Refusing update from remote peer 455a4c2bf6d545feb792f749c645f80d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:44.796020 19419 consensus_queue.cc:1035] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d2ab7402e64e238a26e97181cb56e7" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 36417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:44.796183 19417 consensus_queue.cc:1035] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b66bfa111c164faca8c284f752f5ff1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:44.803746 19424 mvcc.cc:205] Tried to move back new op lower bound from 7334324980914233344 to 7334324980806209536. Current Snapshot: MvccSnapshot[applied={T|T < 7334324980914233344}]
I20260928 14:44:44.804639 19423 mvcc.cc:205] Tried to move back new op lower bound from 7334324980914233344 to 7334324980806209536. Current Snapshot: MvccSnapshot[applied={T|T < 7334324980914233344}]
I20260928 14:44:44.811205 18929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:44.811369 18929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:44.813649 18929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5eb7e563d0854be0bf39f6a45b174e13: Sending DeleteTablet for 3 replicas of tablet f53e8e60aa39496a9ea4c5cef44b249d
I20260928 14:44:44.814424 19192 tablet_service.cc:1514] Processing DeleteTablet for tablet f53e8e60aa39496a9ea4c5cef44b249d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:44 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:36138
I20260928 14:44:44.814428 19057 tablet_service.cc:1514] Processing DeleteTablet for tablet f53e8e60aa39496a9ea4c5cef44b249d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:44 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:37302
I20260928 14:44:44.814877 19433 tablet_replica.cc:330] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: stopping tablet replica
I20260928 14:44:44.815165 19433 raft_consensus.cc:2229] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:44:44.815294 19326 tablet_service.cc:1514] Processing DeleteTablet for tablet f53e8e60aa39496a9ea4c5cef44b249d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:44 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:52182
I20260928 14:44:44.815390 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 18993
I20260928 14:44:44.815514 19433 raft_consensus.cc:2258] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:44.817039 19434 tablet_replica.cc:330] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: stopping tablet replica
I20260928 14:44:44.817325 19434 raft_consensus.cc:2229] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:44.817608 19434 raft_consensus.cc:2258] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:44.818249 19433 ts_tablet_manager.cc:1905] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:44.818867 19434 ts_tablet_manager.cc:1905] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:44.820869 19434 ts_tablet_manager.cc:1918] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:44.820931 19433 ts_tablet_manager.cc:1918] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:44.820977 19434 log.cc:1196] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/f53e8e60aa39496a9ea4c5cef44b249d
I20260928 14:44:44.821017 19433 log.cc:1196] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/f53e8e60aa39496a9ea4c5cef44b249d
I20260928 14:44:44.821272 19434 ts_tablet_manager.cc:1939] T f53e8e60aa39496a9ea4c5cef44b249d P b66bfa111c164faca8c284f752f5ff1f: Deleting consensus metadata
I20260928 14:44:44.821333 19433 ts_tablet_manager.cc:1939] T f53e8e60aa39496a9ea4c5cef44b249d P 455a4c2bf6d545feb792f749c645f80d: Deleting consensus metadata
I20260928 14:44:44.822403 18917 catalog_manager.cc:4841] TS b66bfa111c164faca8c284f752f5ff1f (127.1.201.3:34339): tablet f53e8e60aa39496a9ea4c5cef44b249d (table test-table [id=10558ab09d0e412b8ce6d5e468914a91]) successfully deleted
I20260928 14:44:44.822588 18917 catalog_manager.cc:4841] TS 455a4c2bf6d545feb792f749c645f80d (127.1.201.2:36447): tablet f53e8e60aa39496a9ea4c5cef44b249d (table test-table [id=10558ab09d0e412b8ce6d5e468914a91]) successfully deleted
W20260928 14:44:44.826457 18915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:36417 (error 108)
I20260928 14:44:44.827324 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 19128
I20260928 14:44:44.827518 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:44.827223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:36417 (local address 127.0.0.1:37312)
0928 14:44:44.827342 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:44.827423 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:36417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:44:44.828353 18915 catalog_manager.cc:4584] TS 36d2ab7402e64e238a26e97181cb56e7 (127.1.201.1:36417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f53e8e60aa39496a9ea4c5cef44b249d: Network error: Client connection negotiation failed: client connection to 127.1.201.1:36417: connect: Connection refused (error 111)
I20260928 14:44:44.833212 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 19263
I20260928 14:44:44.838500 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 18899
2026-09-28T14:44:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2251 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T14:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:44Z Disabled control of system clock
I20260928 14:44:44.868469 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42449
--webserver_interface=127.1.201.62
--webserver_port=0
--builtin_ntp_servers=127.1.201.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:44:44.976418 19439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:44.976755 19439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:44.976812 19439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:44.980675 19439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:44.980723 19439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:44.980738 19439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:44.980755 19439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:44.980770 19439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:44.985951 19439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:42449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:44.986907 19439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:44.988190 19439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:44.996524 19445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:44.996549 19439 server_base.cc:1034] running on GCE node
W20260928 14:44:44.996516 19444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:44.996515 19447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:44.997287 19439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:44.998432 19439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:44.999611 19439 hybrid_clock.cc:648] HybridClock initialized: now 1790606684999587 us; error 35 us; skew 500 ppm
I20260928 14:44:45.001299 19439 webserver.cc:460] Webserver started at http://127.1.201.62:46279/ using document root <none> and password file <none>
I20260928 14:44:45.001926 19439 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:45.002013 19439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:45.002236 19439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:45.004254 19439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "ef66eaefbfc44717b67f863e8779d76b"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "9c02962224a76661f08a93a88148ddef"
server_key_iv: "3f40688c27aec1ff8ae9384d25a09dcc"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.004822 19439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "ef66eaefbfc44717b67f863e8779d76b"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "9c02962224a76661f08a93a88148ddef"
server_key_iv: "3f40688c27aec1ff8ae9384d25a09dcc"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.008373 19439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:44:45.010645 19453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:45.011814 19439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:45.011946 19439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "ef66eaefbfc44717b67f863e8779d76b"
format_stamp: "Formatted at 2026-09-28 14:44:44 on dist-test-slave-0n27"
server_key: "9c02962224a76661f08a93a88148ddef"
server_key_iv: "3f40688c27aec1ff8ae9384d25a09dcc"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.012054 19439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:45.029482 19439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:45.030150 19439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:45.030359 19439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:45.037484 19439 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:42449
I20260928 14:44:45.037508 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:42449 every 8 connection(s)
I20260928 14:44:45.038419 19439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:45.041401 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:45.043759 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 19439
I20260928 14:44:45.043895 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:45.044219 1828 external_mini_cluster.cc:1428] Setting key b628bc080e8d4c4bdaa0b982ab62f7c5
I20260928 14:44:45.048255 19506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b: Bootstrap starting.
I20260928 14:44:45.050477 19506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:45.051324 19506 log.cc:824] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:45.052955 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.045472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.1.201.62:42449)
0928 14:44:45.046087 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.046102 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.046136 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:44:45.046781 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.046796 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.047175 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.047502 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:45.047921 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.048896 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.049813 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.050269 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.050445 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.051290 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:45.051295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:45.051307 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:45.051710 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:45.051765 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:45.051784 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:45.051909 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.052053 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.052301 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
I20260928 14:44:45.053220 19506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b: No bootstrap required, opened a new log
I20260928 14:44:45.057129 19506 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66eaefbfc44717b67f863e8779d76b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42449 } }
I20260928 14:44:45.057360 19506 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:45.057446 19506 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef66eaefbfc44717b67f863e8779d76b, State: Initialized, Role: FOLLOWER
I20260928 14:44:45.057983 19506 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66eaefbfc44717b67f863e8779d76b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42449 } }
I20260928 14:44:45.058149 19506 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:45.058243 19506 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:45.058338 19506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:45.059175 19506 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66eaefbfc44717b67f863e8779d76b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42449 } }
I20260928 14:44:45.059504 19506 leader_election.cc:302] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef66eaefbfc44717b67f863e8779d76b; no voters:
I20260928 14:44:45.059801 19506 leader_election.cc:288] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:45.059901 19511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:45.060150 19511 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [term 1 LEADER]: Becoming Leader. State: Replica: ef66eaefbfc44717b67f863e8779d76b, State: Running, Role: LEADER
I20260928 14:44:45.060441 19511 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66eaefbfc44717b67f863e8779d76b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42449 } }
I20260928 14:44:45.061013 19506 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:45.062040 19513 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef66eaefbfc44717b67f863e8779d76b. Latest consensus state: current_term: 1 leader_uuid: "ef66eaefbfc44717b67f863e8779d76b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66eaefbfc44717b67f863e8779d76b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42449 } } }
I20260928 14:44:45.062049 19512 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef66eaefbfc44717b67f863e8779d76b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66eaefbfc44717b67f863e8779d76b" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42449 } } }
I20260928 14:44:45.062196 19512 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:45.062196 19513 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:45.062697 19520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:45.065810 19520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:45.071410 19520 catalog_manager.cc:1347] Generated new cluster ID: 86b0b32f4b314a28baf5e30ca1e20993
I20260928 14:44:45.071480 19520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:45.082875 19520 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:45.083665 19520 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:45.092623 19520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b: Generated new TSK 0
I20260928 14:44:45.093277 19520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:44:45.104544 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42449
--builtin_ntp_servers=127.1.201.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:44:45.214346 19530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:45.214614 19530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:45.214687 19530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:45.218828 19530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:45.218902 19530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:45.218989 19530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:44:45.223707 19530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:45.224738 19530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:45.226142 19530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:45.235427 19535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:45.235473 19530 server_base.cc:1034] running on GCE node
W20260928 14:44:45.235437 19536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:45.235638 19538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:45.236049 19530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:45.236645 19530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:45.237845 19530 hybrid_clock.cc:648] HybridClock initialized: now 1790606685237831 us; error 23 us; skew 500 ppm
I20260928 14:44:45.239533 19530 webserver.cc:460] Webserver started at http://127.1.201.1:45999/ using document root <none> and password file <none>
I20260928 14:44:45.240092 19530 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:45.240180 19530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:45.240381 19530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:45.242439 19530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "325c4ac760b14f13a1bf42c5c2cf940f"
format_stamp: "Formatted at 2026-09-28 14:44:45 on dist-test-slave-0n27"
server_key: "9cbf9dcfab7e6cd50c2f27456b060760"
server_key_iv: "3351771049847818a8f401cb2da3502a"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.242981 19530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "325c4ac760b14f13a1bf42c5c2cf940f"
format_stamp: "Formatted at 2026-09-28 14:44:45 on dist-test-slave-0n27"
server_key: "9cbf9dcfab7e6cd50c2f27456b060760"
server_key_iv: "3351771049847818a8f401cb2da3502a"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.246469 19530 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:44:45.248729 19544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:45.249898 19530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 14:44:45.250018 19530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "325c4ac760b14f13a1bf42c5c2cf940f"
format_stamp: "Formatted at 2026-09-28 14:44:45 on dist-test-slave-0n27"
server_key: "9cbf9dcfab7e6cd50c2f27456b060760"
server_key_iv: "3351771049847818a8f401cb2da3502a"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.250123 19530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:45.262305 19530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:45.262895 19530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:45.263065 19530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:45.263621 19530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:45.264464 19530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:45.264529 19530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:45.264590 19530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:45.264637 19530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:45.272991 19530 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:35085
I20260928 14:44:45.273020 19658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:35085 every 8 connection(s)
I20260928 14:44:45.273904 19530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:44:45.280856 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 19530
I20260928 14:44:45.281001 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:44:45.281293 1828 external_mini_cluster.cc:1428] Setting key b695b7e5815446ff26050d6f412c2d4a
I20260928 14:44:45.282996 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.275549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:52257 (local address 127.1.201.62:42449)
0928 14:44:45.275703 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.275706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.276626 (+ 920us) server_negotiation.cc:407] Connection header received
0928 14:44:45.278137 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.278140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.278198 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.278306 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:45.279163 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.279707 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.280871 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.281207 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.281327 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.282516 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:45.282519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:45.282523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:45.282535 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:45.282555 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:45.282570 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:45.282648 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.282804 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.282865 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:44:45.283375 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42449
--builtin_ntp_servers=127.1.201.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:44:45.283874 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.275878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42449 (local address 127.1.201.1:52257)
0928 14:44:45.276482 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:45.276520 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:45.277935 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:45.278476 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:45.278482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:45.278565 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:45.279027 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.279038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.279845 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.279853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.280745 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.280754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.281368 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.281372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.281576 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.282231 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:44:45.282233 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:45.282285 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:44:45.282362 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:45.282649 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:45.282659 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:45.282667 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:44:45.282804 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 14:44:45.283077 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260928 14:44:45.285145 19659 heartbeater.cc:347] Connected to a master server at 127.1.201.62:42449
I20260928 14:44:45.285364 19659 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:45.285907 19659 heartbeater.cc:511] Master 127.1.201.62:42449 requested a full tablet report, sending...
I20260928 14:44:45.287413 19469 ts_manager.cc:194] Registered new tserver with Master: 325c4ac760b14f13a1bf42c5c2cf940f (127.1.201.1:35085)
I20260928 14:44:45.289132 19469 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.1:52257
W20260928 14:44:45.396320 19662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:45.396552 19662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:45.396595 19662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:45.400593 19662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:45.400645 19662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:45.400701 19662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:44:45.405416 19662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:45.406425 19662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:45.407706 19662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:45.417016 19670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:45.417014 19667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:45.417091 19662 server_base.cc:1034] running on GCE node
W20260928 14:44:45.417040 19668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:45.417714 19662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:45.418322 19662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:45.419495 19662 hybrid_clock.cc:648] HybridClock initialized: now 1790606685419478 us; error 29 us; skew 500 ppm
I20260928 14:44:45.421123 19662 webserver.cc:460] Webserver started at http://127.1.201.2:33195/ using document root <none> and password file <none>
I20260928 14:44:45.421662 19662 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:45.421746 19662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:45.421965 19662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:45.423934 19662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "326c98ed0b304945ae648ace4342af1d"
format_stamp: "Formatted at 2026-09-28 14:44:45 on dist-test-slave-0n27"
server_key: "4e51e29c4afc501512707f2991d3553b"
server_key_iv: "c0ac86198517cb53a7748c5d586e00e0"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.424475 19662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "326c98ed0b304945ae648ace4342af1d"
format_stamp: "Formatted at 2026-09-28 14:44:45 on dist-test-slave-0n27"
server_key: "4e51e29c4afc501512707f2991d3553b"
server_key_iv: "c0ac86198517cb53a7748c5d586e00e0"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.427904 19662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:44:45.430246 19676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:45.431370 19662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:45.431491 19662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "326c98ed0b304945ae648ace4342af1d"
format_stamp: "Formatted at 2026-09-28 14:44:45 on dist-test-slave-0n27"
server_key: "4e51e29c4afc501512707f2991d3553b"
server_key_iv: "c0ac86198517cb53a7748c5d586e00e0"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.431594 19662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:45.457203 19662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:45.457942 19662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:45.458132 19662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:45.458707 19662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:45.459666 19662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:45.459744 19662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:45.459833 19662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:45.459883 19662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:45.468407 19662 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:46113
I20260928 14:44:45.468453 19789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:46113 every 8 connection(s)
I20260928 14:44:45.469272 19662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:44:45.470604 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 19662
I20260928 14:44:45.470768 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:44:45.471095 1828 external_mini_cluster.cc:1428] Setting key 647bc8b660d67a3f385a5503bbf97f11
I20260928 14:44:45.473147 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42449
--builtin_ntp_servers=127.1.201.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:44:45.479691 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.471168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:47447 (local address 127.1.201.62:42449)
0928 14:44:45.471337 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.471342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.472511 (+ 1169us) server_negotiation.cc:407] Connection header received
0928 14:44:45.474370 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.474373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.474420 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.474484 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:45.475470 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.476303 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.477465 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.477927 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.478052 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.479104 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:45.479110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:45.479115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:45.479133 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:45.479163 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:45.479177 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:45.479268 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.479409 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.479486 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:44:45.480429 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.471638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42449 (local address 127.1.201.2:47447)
0928 14:44:45.472361 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:45.472410 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:45.474157 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:45.474682 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:45.474694 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:45.474814 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:45.475325 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.475337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.476428 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.476430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.477339 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.477348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.478088 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.478091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.478229 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.478850 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:44:45.478852 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:45.478889 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:44:45.478947 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:45.479271 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:45.479277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:45.479280 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:44:45.479371 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:44:45.479570 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":132,"threads_started":1}
I20260928 14:44:45.481618 19790 heartbeater.cc:347] Connected to a master server at 127.1.201.62:42449
I20260928 14:44:45.481870 19790 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:45.482343 19790 heartbeater.cc:511] Master 127.1.201.62:42449 requested a full tablet report, sending...
I20260928 14:44:45.483361 19469 ts_manager.cc:194] Registered new tserver with Master: 326c98ed0b304945ae648ace4342af1d (127.1.201.2:46113)
I20260928 14:44:45.484208 19469 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.2:47447
W20260928 14:44:45.586012 19793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:45.586242 19793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:45.586278 19793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:45.590168 19793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:45.590214 19793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:45.590263 19793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:44:45.595220 19793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:45.596179 19793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:45.597465 19793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:45.606808 19799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:45.606827 19798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:45.606913 19793 server_base.cc:1034] running on GCE node
W20260928 14:44:45.606861 19801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:45.607475 19793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:45.608019 19793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:45.609177 19793 hybrid_clock.cc:648] HybridClock initialized: now 1790606685609155 us; error 36 us; skew 500 ppm
I20260928 14:44:45.610935 19793 webserver.cc:460] Webserver started at http://127.1.201.3:38103/ using document root <none> and password file <none>
I20260928 14:44:45.611483 19793 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:45.611541 19793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:45.611771 19793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:45.613943 19793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "527e0f8ac01b43be9e39a604f566c17a"
format_stamp: "Formatted at 2026-09-28 14:44:45 on dist-test-slave-0n27"
server_key: "78da7898894551cc0af3e54edfc92fc8"
server_key_iv: "9ce052f97f4038fff44f963d8f9b2f8d"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.614476 19793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "527e0f8ac01b43be9e39a604f566c17a"
format_stamp: "Formatted at 2026-09-28 14:44:45 on dist-test-slave-0n27"
server_key: "78da7898894551cc0af3e54edfc92fc8"
server_key_iv: "9ce052f97f4038fff44f963d8f9b2f8d"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.618089 19793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:44:45.620362 19807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:45.621490 19793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:45.621721 19793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "527e0f8ac01b43be9e39a604f566c17a"
format_stamp: "Formatted at 2026-09-28 14:44:45 on dist-test-slave-0n27"
server_key: "78da7898894551cc0af3e54edfc92fc8"
server_key_iv: "9ce052f97f4038fff44f963d8f9b2f8d"
server_key_version: "encryptionkey@0"
I20260928 14:44:45.621877 19793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:45.636901 19793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:45.637508 19793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:45.637692 19793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:45.638255 19793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:45.639165 19793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:45.639211 19793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:45.639278 19793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:45.639336 19793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:45.647549 19793 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:41445
I20260928 14:44:45.647569 19920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:41445 every 8 connection(s)
I20260928 14:44:45.648396 19793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:44:45.649669 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 19793
I20260928 14:44:45.649813 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:44:45.650159 1828 external_mini_cluster.cc:1428] Setting key 52f052b2a36f7be620d9cf64f5e305e2
I20260928 14:44:45.657670 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.650366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:52171 (local address 127.1.201.62:42449)
0928 14:44:45.650504 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.650509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.651816 (+ 1307us) server_negotiation.cc:407] Connection header received
0928 14:44:45.653322 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.653325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.653371 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.653450 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:45.654462 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.654987 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.656069 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.656287 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.656358 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.657281 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:45.657284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:45.657286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:45.657297 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:45.657317 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:45.657325 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:45.657379 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.657508 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.657555 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:44:45.658478 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.650848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:42449 (local address 127.1.201.3:52171)
0928 14:44:45.651649 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:45.651714 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:45.653127 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:45.653591 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:45.653597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:45.653690 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:45.654326 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.654340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.655093 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.655095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.655966 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.655975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.656404 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.656406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.656539 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.657059 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:44:45.657061 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:45.657096 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:44:45.657153 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:45.657395 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:45.657401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:45.657405 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:44:45.657490 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:44:45.657722 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":135,"threads_started":1}
I20260928 14:44:45.659510 19921 heartbeater.cc:347] Connected to a master server at 127.1.201.62:42449
I20260928 14:44:45.659736 19921 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:45.660223 19921 heartbeater.cc:511] Master 127.1.201.62:42449 requested a full tablet report, sending...
I20260928 14:44:45.661230 19469 ts_manager.cc:194] Registered new tserver with Master: 527e0f8ac01b43be9e39a604f566c17a (127.1.201.3:41445)
I20260928 14:44:45.661855 19469 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.3:52171
I20260928 14:44:45.663985 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:44:45.669587 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.665831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.1.201.62:42449)
0928 14:44:45.665934 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.665937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.666188 (+ 251us) server_negotiation.cc:407] Connection header received
0928 14:44:45.666347 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.666352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.666399 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.666481 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:45.666775 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.667264 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.668081 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.668299 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.668407 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.669203 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:45.669207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:45.669210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:45.669228 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:45.669244 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:45.669252 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:45.669327 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.669413 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.669461 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:44:45.672024 19469 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:44:45.674932 19469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:44:45.689080 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.682828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:41445 (local address 127.0.0.1:54242)
0928 14:44:45.683566 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:45.683613 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:45.683743 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:45.684907 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:45.684910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:45.684939 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:45.685082 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.685088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.686509 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.686513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.687557 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.687567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.688538 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.688542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.688651 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.688686 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 14:44:45.688788 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:44:45.688913 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":74,"spinlock_wait_cycles":4480,"thread_start_us":76,"threads_started":1}
I20260928 14:44:45.689244 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.682997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35085 (local address 127.0.0.1:59606)
0928 14:44:45.684046 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:45.684060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:45.684150 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:45.684905 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:45.684908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:45.684937 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:45.685082 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.685088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.686624 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.686627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.687851 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.687859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.688785 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.688789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.688870 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.688892 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:44:45.688967 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:44:45.689039 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":92,"spinlock_wait_cycles":5248,"thread_start_us":88,"threads_started":1}
I20260928 14:44:45.689647 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.683599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.1.201.1:35085)
0928 14:44:45.684036 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.684053 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.684265 (+ 212us) server_negotiation.cc:407] Connection header received
0928 14:44:45.684351 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.684358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.684554 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.684721 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:45.685206 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.686523 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.687983 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.688527 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.688624 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.688818 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.689013 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.689115 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":139,"threads_started":1}
I20260928 14:44:45.689677 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.682974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.1.201.3:41445)
0928 14:44:45.684464 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.684471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.684488 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:44:45.684542 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.684545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.684735 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.684911 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:45.685201 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.686382 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.687685 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.688409 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.688519 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.688701 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.689431 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.689523 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1409,"thread_start_us":139,"threads_started":1}
I20260928 14:44:45.689869 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.683883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.1.201.2:46113)
0928 14:44:45.684314 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.684325 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.684345 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:44:45.684405 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.684409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.684553 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.684721 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:45.685231 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.686047 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.687953 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.688689 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.688800 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.688966 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.689584 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.689688 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":123,"threads_started":1}
I20260928 14:44:45.689883 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.683464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:46113 (local address 127.0.0.1:56346)
0928 14:44:45.684186 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:45.684197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:45.684271 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:45.684740 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:45.684747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:45.684767 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:45.684911 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.684918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.686171 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.686181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.687827 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.687838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.689023 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.689026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.689222 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.689245 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:44:45.689322 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:44:45.689406 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":21,"thread_start_us":100,"threads_started":1}
I20260928 14:44:45.692384 19724 tablet_service.cc:1467] Processing CreateTablet for tablet bf01ca971cbe490ab914d3d85ad3c08b (DEFAULT_TABLE table=test-table [id=6b742cb3adaf43ba9eff229178262e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:45.692559 19855 tablet_service.cc:1467] Processing CreateTablet for tablet bf01ca971cbe490ab914d3d85ad3c08b (DEFAULT_TABLE table=test-table [id=6b742cb3adaf43ba9eff229178262e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:45.692559 19592 tablet_service.cc:1467] Processing CreateTablet for tablet bf01ca971cbe490ab914d3d85ad3c08b (DEFAULT_TABLE table=test-table [id=6b742cb3adaf43ba9eff229178262e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:45.693385 19724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf01ca971cbe490ab914d3d85ad3c08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:45.693542 19855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf01ca971cbe490ab914d3d85ad3c08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:45.693542 19592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf01ca971cbe490ab914d3d85ad3c08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:45.699492 19938 tablet_bootstrap.cc:492] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: Bootstrap starting.
I20260928 14:44:45.701164 19939 tablet_bootstrap.cc:492] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f: Bootstrap starting.
I20260928 14:44:45.701689 19938 tablet_bootstrap.cc:654] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:45.702401 19938 log.cc:824] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:45.702984 19940 tablet_bootstrap.cc:492] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: Bootstrap starting.
I20260928 14:44:45.702989 19939 tablet_bootstrap.cc:654] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:45.703796 19939 log.cc:824] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:45.704489 19938 tablet_bootstrap.cc:492] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: No bootstrap required, opened a new log
I20260928 14:44:45.704734 19938 ts_tablet_manager.cc:1397] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:44:45.704988 19940 tablet_bootstrap.cc:654] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:45.705752 19940 log.cc:824] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:45.705922 19939 tablet_bootstrap.cc:492] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f: No bootstrap required, opened a new log
I20260928 14:44:45.706103 19939 ts_tablet_manager.cc:1397] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:44:45.707621 19940 tablet_bootstrap.cc:492] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: No bootstrap required, opened a new log
I20260928 14:44:45.707799 19940 ts_tablet_manager.cc:1397] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:44:45.708245 19938 raft_consensus.cc:348] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527e0f8ac01b43be9e39a604f566c17a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41445 } } peers { permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 } } peers { permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 } }
I20260928 14:44:45.708477 19938 raft_consensus.cc:374] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:45.708549 19938 raft_consensus.cc:729] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326c98ed0b304945ae648ace4342af1d, State: Initialized, Role: FOLLOWER
I20260928 14:44:45.709002 19938 consensus_queue.cc:260] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527e0f8ac01b43be9e39a604f566c17a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41445 } } peers { permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 } } peers { permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 } }
I20260928 14:44:45.709630 19790 heartbeater.cc:503] Master 127.1.201.62:42449 was elected leader, sending a full tablet report...
I20260928 14:44:45.709961 19938 ts_tablet_manager.cc:1428] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:44:45.711315 19939 raft_consensus.cc:348] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527e0f8ac01b43be9e39a604f566c17a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41445 } } peers { permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 } } peers { permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 } }
I20260928 14:44:45.711596 19939 raft_consensus.cc:374] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:45.711655 19939 raft_consensus.cc:729] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325c4ac760b14f13a1bf42c5c2cf940f, State: Initialized, Role: FOLLOWER
I20260928 14:44:45.712044 19939 consensus_queue.cc:260] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527e0f8ac01b43be9e39a604f566c17a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41445 } } peers { permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 } } peers { permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 } }
I20260928 14:44:45.712031 19940 raft_consensus.cc:348] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527e0f8ac01b43be9e39a604f566c17a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41445 } } peers { permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 } } peers { permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 } }
I20260928 14:44:45.712291 19940 raft_consensus.cc:374] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:45.712396 19940 raft_consensus.cc:729] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527e0f8ac01b43be9e39a604f566c17a, State: Initialized, Role: FOLLOWER
I20260928 14:44:45.712656 19659 heartbeater.cc:503] Master 127.1.201.62:42449 was elected leader, sending a full tablet report...
I20260928 14:44:45.712844 19940 consensus_queue.cc:260] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527e0f8ac01b43be9e39a604f566c17a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41445 } } peers { permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 } } peers { permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 } }
I20260928 14:44:45.712966 19939 ts_tablet_manager.cc:1428] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:44:45.713716 19940 ts_tablet_manager.cc:1428] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:44:45.714097 19921 heartbeater.cc:503] Master 127.1.201.62:42449 was elected leader, sending a full tablet report...
W20260928 14:44:45.720098 19791 tablet.cc:2369] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:45.739697 19946 raft_consensus.cc:482] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:45.739892 19946 raft_consensus.cc:504] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527e0f8ac01b43be9e39a604f566c17a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41445 } } peers { permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 } } peers { permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 } }
I20260928 14:44:45.741254 19946 leader_election.cc:288] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 325c4ac760b14f13a1bf42c5c2cf940f (127.1.201.1:35085), 326c98ed0b304945ae648ace4342af1d (127.1.201.2:46113)
I20260928 14:44:45.745422 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.741457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35085 (local address 127.1.201.3:39703)
0928 14:44:45.741661 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:45.741676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:45.741799 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:45.742219 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:45.742222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:45.742263 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:45.742378 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.742387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.743378 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.743384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.744164 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.744173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.744960 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.744963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.745076 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.745100 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:44:45.745209 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:44:45.745295 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 14:44:45.745503 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.741552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:39703 (local address 127.1.201.1:35085)
0928 14:44:45.741689 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.741694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.741713 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:44:45.741915 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.741921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.741999 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.742097 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:45.742524 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.743295 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.744286 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.744833 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.744949 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.745010 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.745307 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.745383 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:44:45.746126 19612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf01ca971cbe490ab914d3d85ad3c08b" candidate_uuid: "527e0f8ac01b43be9e39a604f566c17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" is_pre_election: true
I20260928 14:44:45.746470 19612 raft_consensus.cc:2454] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527e0f8ac01b43be9e39a604f566c17a in term 0.
I20260928 14:44:45.746902 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.742066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:46113 (local address 127.1.201.3:40789)
0928 14:44:45.742431 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:45.742447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:45.742577 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:45.742912 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:45.742915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:45.742932 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:45.743036 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.743042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.744208 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.744215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.745393 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:45.745406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.746434 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.746437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:45.746546 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.746568 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:44:45.746657 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:44:45.746726 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":124,"threads_started":1}
I20260928 14:44:45.746948 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.741973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:40789 (local address 127.1.201.2:46113)
0928 14:44:45.742098 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.742102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.742512 (+ 410us) server_negotiation.cc:407] Connection header received
0928 14:44:45.742692 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.742695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.742743 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.742791 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:45.743175 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.744034 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.745554 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.746173 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.746260 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.746315 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.746711 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.746812 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:44:45.747069 19809 leader_election.cc:302] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 325c4ac760b14f13a1bf42c5c2cf940f, 527e0f8ac01b43be9e39a604f566c17a; no voters:
I20260928 14:44:45.747327 19946 raft_consensus.cc:2781] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:44:45.747437 19946 raft_consensus.cc:482] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:44:45.747504 19946 raft_consensus.cc:3037] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:45.747756 19744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf01ca971cbe490ab914d3d85ad3c08b" candidate_uuid: "527e0f8ac01b43be9e39a604f566c17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c98ed0b304945ae648ace4342af1d" is_pre_election: true
I20260928 14:44:45.748034 19744 raft_consensus.cc:2454] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527e0f8ac01b43be9e39a604f566c17a in term 0.
I20260928 14:44:45.748863 19946 raft_consensus.cc:504] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527e0f8ac01b43be9e39a604f566c17a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41445 } } peers { permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 } } peers { permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 } }
I20260928 14:44:45.749243 19946 leader_election.cc:288] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [CANDIDATE]: Term 1 election: Requested vote from peers 325c4ac760b14f13a1bf42c5c2cf940f (127.1.201.1:35085), 326c98ed0b304945ae648ace4342af1d (127.1.201.2:46113)
I20260928 14:44:45.749630 19744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf01ca971cbe490ab914d3d85ad3c08b" candidate_uuid: "527e0f8ac01b43be9e39a604f566c17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c98ed0b304945ae648ace4342af1d"
I20260928 14:44:45.749678 19612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf01ca971cbe490ab914d3d85ad3c08b" candidate_uuid: "527e0f8ac01b43be9e39a604f566c17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "325c4ac760b14f13a1bf42c5c2cf940f"
I20260928 14:44:45.749763 19744 raft_consensus.cc:3037] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:45.749818 19612 raft_consensus.cc:3037] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:45.750794 19612 raft_consensus.cc:2454] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527e0f8ac01b43be9e39a604f566c17a in term 1.
I20260928 14:44:45.751122 19809 leader_election.cc:302] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 325c4ac760b14f13a1bf42c5c2cf940f, 527e0f8ac01b43be9e39a604f566c17a; no voters:
I20260928 14:44:45.751264 19744 raft_consensus.cc:2454] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527e0f8ac01b43be9e39a604f566c17a in term 1.
I20260928 14:44:45.751314 19946 raft_consensus.cc:2781] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:45.751507 19946 raft_consensus.cc:686] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 1 LEADER]: Becoming Leader. State: Replica: 527e0f8ac01b43be9e39a604f566c17a, State: Running, Role: LEADER
I20260928 14:44:45.751907 19946 consensus_queue.cc:237] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527e0f8ac01b43be9e39a604f566c17a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41445 } } peers { permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 } } peers { permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 } }
I20260928 14:44:45.755628 19469 catalog_manager.cc:5495] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a reported cstate change: term changed from 0 to 1, leader changed from <none> to 527e0f8ac01b43be9e39a604f566c17a (127.1.201.3). New cstate: current_term: 1 leader_uuid: "527e0f8ac01b43be9e39a604f566c17a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527e0f8ac01b43be9e39a604f566c17a" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 41445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 } health_report { overall_health: UNKNOWN } } }
I20260928 14:44:45.770491 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.765586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.1.201.3:41445)
0928 14:44:45.765729 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:45.765736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:45.765754 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:44:45.765938 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:45.765943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:45.766018 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:45.766134 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:44:45.766400 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.767275 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.768610 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:45.769028 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:45.769152 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:45.769468 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:44:45.770075 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:44:45.770208 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:45.770246 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 14:44:45.770310 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 14:44:45.775063 19660 tablet.cc:2369] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:45.777076 19612 raft_consensus.cc:1261] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f [term 1 FOLLOWER]: Refusing update from remote peer 527e0f8ac01b43be9e39a604f566c17a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:45.777073 19744 raft_consensus.cc:1261] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d [term 1 FOLLOWER]: Refusing update from remote peer 527e0f8ac01b43be9e39a604f566c17a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:45.777750 19946 consensus_queue.cc:1035] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [LEADER]: Connected to new peer: Peer: permanent_uuid: "326c98ed0b304945ae648ace4342af1d" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 46113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:45.778003 19948 consensus_queue.cc:1035] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [LEADER]: Connected to new peer: Peer: permanent_uuid: "325c4ac760b14f13a1bf42c5c2cf940f" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:45.784971 19951 mvcc.cc:205] Tried to move back new op lower bound from 7334324984935866368 to 7334324984843124736. Current Snapshot: MvccSnapshot[applied={T|T < 7334324984935866368}]
I20260928 14:44:45.786288 19952 mvcc.cc:205] Tried to move back new op lower bound from 7334324984935866368 to 7334324984843124736. Current Snapshot: MvccSnapshot[applied={T|T < 7334324984935866368}]
I20260928 14:44:45.793617 19470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:45.793788 19470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:45.796100 19470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef66eaefbfc44717b67f863e8779d76b: Sending DeleteTablet for 3 replicas of tablet bf01ca971cbe490ab914d3d85ad3c08b
I20260928 14:44:45.796784 19855 tablet_service.cc:1514] Processing DeleteTablet for tablet bf01ca971cbe490ab914d3d85ad3c08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:45 UTC) from {username='slave'} at 127.0.0.1:54242
I20260928 14:44:45.796988 19592 tablet_service.cc:1514] Processing DeleteTablet for tablet bf01ca971cbe490ab914d3d85ad3c08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:45 UTC) from {username='slave'} at 127.0.0.1:59606
I20260928 14:44:45.797374 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 19530
I20260928 14:44:45.797431 19962 tablet_replica.cc:330] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f: stopping tablet replica
I20260928 14:44:45.797437 19724 tablet_service.cc:1514] Processing DeleteTablet for tablet bf01ca971cbe490ab914d3d85ad3c08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:45 UTC) from {username='slave'} at 127.0.0.1:56346
I20260928 14:44:45.797734 19962 raft_consensus.cc:2229] T bf01ca971cbe490ab914d3d85ad3c08b P 325c4ac760b14f13a1bf42c5c2cf940f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:45.798118 19963 tablet_replica.cc:330] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: stopping tablet replica
I20260928 14:44:45.798417 19963 raft_consensus.cc:2229] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:45.798692 19963 raft_consensus.cc:2258] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:45.798902 19961 tablet_replica.cc:330] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: stopping tablet replica
I20260928 14:44:45.799223 19961 raft_consensus.cc:2229] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:44:45.799592 19961 raft_consensus.cc:2258] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:45.801031 19963 ts_tablet_manager.cc:1905] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:45.802316 19961 ts_tablet_manager.cc:1905] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:45.804592 19963 ts_tablet_manager.cc:1918] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:45.804755 19963 log.cc:1196] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/bf01ca971cbe490ab914d3d85ad3c08b
I20260928 14:44:45.804912 19961 ts_tablet_manager.cc:1918] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:45.805127 19963 ts_tablet_manager.cc:1939] T bf01ca971cbe490ab914d3d85ad3c08b P 326c98ed0b304945ae648ace4342af1d: Deleting consensus metadata
I20260928 14:44:45.805289 19961 log.cc:1196] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/bf01ca971cbe490ab914d3d85ad3c08b
I20260928 14:44:45.805678 19961 ts_tablet_manager.cc:1939] T bf01ca971cbe490ab914d3d85ad3c08b P 527e0f8ac01b43be9e39a604f566c17a: Deleting consensus metadata
I20260928 14:44:45.806398 19455 catalog_manager.cc:4841] TS 326c98ed0b304945ae648ace4342af1d (127.1.201.2:46113): tablet bf01ca971cbe490ab914d3d85ad3c08b (table test-table [id=6b742cb3adaf43ba9eff229178262e0a]) successfully deleted
I20260928 14:44:45.806738 19456 catalog_manager.cc:4841] TS 527e0f8ac01b43be9e39a604f566c17a (127.1.201.3:41445): tablet bf01ca971cbe490ab914d3d85ad3c08b (table test-table [id=6b742cb3adaf43ba9eff229178262e0a]) successfully deleted
W20260928 14:44:45.807695 19455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:35085 (error 108)
I20260928 14:44:45.807870 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 19662
I20260928 14:44:45.809327 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:45.808988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35085 (local address 127.0.0.1:59616)
0928 14:44:45.809171 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:45.809243 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:35085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260928 14:44:45.809468 19455 catalog_manager.cc:4584] TS 325c4ac760b14f13a1bf42c5c2cf940f (127.1.201.1:35085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf01ca971cbe490ab914d3d85ad3c08b: Network error: Client connection negotiation failed: client connection to 127.1.201.1:35085: connect: Connection refused (error 111)
I20260928 14:44:45.813377 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 19793
I20260928 14:44:45.818298 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 19439
2026-09-28T14:44:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (978 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:44:45 dist-test-slave-0n27 krb5kdc[19967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:44:45 dist-test-slave-0n27 krb5kdc[19967](info): set up 2 sockets
Sep 28 14:44:45 dist-test-slave-0n27 krb5kdc[19967](info): commencing operation
krb5kdc: starting...
W20260928 14:44:46.830549 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:44:46 dist-test-slave-0n27 krb5kdc[19967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:44:46.946120 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:34511
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:47.053529 19980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:47.053768 19980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:47.053877 19980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:47.057709 19980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:47.057754 19980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:47.057773 19980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:47.057885 19980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:47.057901 19980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:47.063140 19980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:34511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:47.064046 19980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:47.064945 19980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:47.072906 19986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:47.072978 19985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:47.072960 19980 server_base.cc:1034] running on GCE node
W20260928 14:44:47.073107 19988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:47.073732 19980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:47.074796 19980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:47.075981 19980 hybrid_clock.cc:648] HybridClock initialized: now 1790606687075969 us; error 25 us; skew 500 ppm
Sep 28 14:44:47 dist-test-slave-0n27 krb5kdc[19967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606687, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:47.078760 19980 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:44:47.079566 19980 webserver.cc:460] Webserver started at http://127.1.201.62:39079/ using document root <none> and password file <none>
I20260928 14:44:47.080116 19980 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:47.080204 19980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:47.080446 19980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:47.082567 19980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "9d3bb8b93bfe464c04e4291aeace220f"
server_key_iv: "63c732bb6e536108ccf62b61f6740a68"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.083114 19980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "9d3bb8b93bfe464c04e4291aeace220f"
server_key_iv: "63c732bb6e536108ccf62b61f6740a68"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.086648 19980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:44:47.088886 19995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:47.090098 19980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:47.090230 19980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "9d3bb8b93bfe464c04e4291aeace220f"
server_key_iv: "63c732bb6e536108ccf62b61f6740a68"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.090338 19980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:47.136168 19980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:47.139592 19980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:47.139815 19980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:47.146884 19980 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:34511
I20260928 14:44:47.146911 20047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:34511 every 8 connection(s)
I20260928 14:44:47.147825 19980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:47.150550 20048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:47.152181 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 19980
I20260928 14:44:47.152321 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:47.152613 1828 external_mini_cluster.cc:1428] Setting key b711929311d46c662ece0330c0e40825
I20260928 14:44:47.156633 20048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0: Bootstrap starting.
I20260928 14:44:47.158833 20048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:47.159531 20048 log.cc:824] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:44:47 dist-test-slave-0n27 krb5kdc[19967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:44:47.161561 20048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0: No bootstrap required, opened a new log
I20260928 14:44:47.165388 20048 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34511 } }
I20260928 14:44:47.165621 20048 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:47.165695 20048 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a57007cf7143ad8dc9f4d5fc1c7cf0, State: Initialized, Role: FOLLOWER
I20260928 14:44:47.166177 20048 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34511 } }
I20260928 14:44:47.166294 20048 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:47.166378 20048 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:47.166502 20048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:47.166651 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.154052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.1.201.62:34511)
0928 14:44:47.154466 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:47.154480 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:47.154513 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:44:47.155152 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:47.155175 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:47.155480 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:47.155837 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:47.156598 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.157595 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.158509 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.159032 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.159223 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.161892 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:47.161897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:47.161909 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:47.161934 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:47.163621 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:47.164072 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:47.164076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:47.164080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:47.164192 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:47.164468 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:47.164470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:47.164471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:47.164842 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:47.164987 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:47.165285 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 14:44:47.165589 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":133,"threads_started":1}
I20260928 14:44:47.167411 20048 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34511 } }
I20260928 14:44:47.167739 20048 leader_election.cc:302] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a57007cf7143ad8dc9f4d5fc1c7cf0; no voters:
I20260928 14:44:47.168046 20048 leader_election.cc:288] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:47.168174 20053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:47.168371 20053 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [term 1 LEADER]: Becoming Leader. State: Replica: d1a57007cf7143ad8dc9f4d5fc1c7cf0, State: Running, Role: LEADER
I20260928 14:44:47.168757 20053 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34511 } }
I20260928 14:44:47.169248 20048 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:47.169934 20053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a57007cf7143ad8dc9f4d5fc1c7cf0. Latest consensus state: current_term: 1 leader_uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34511 } } }
I20260928 14:44:47.170116 20053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:47.170534 20062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:47.172328 20054 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a57007cf7143ad8dc9f4d5fc1c7cf0" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 34511 } } }
I20260928 14:44:47.172474 20054 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:47.174242 20062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:47.179499 20062 catalog_manager.cc:1347] Generated new cluster ID: 9bd1d2abc933491499e5085a70a04ebb
I20260928 14:44:47.179570 20062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:47.197324 20062 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:47.198256 20062 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:47.204478 20062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0: Generated new TSK 0
I20260928 14:44:47.205106 20062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:44:47.232963 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34511
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:47.342768 20074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:47.343003 20074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:47.343075 20074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:47.347018 20074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:47.347071 20074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:47.347177 20074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:44:47.352324 20074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:47.353294 20074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:47.354286 20074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:47.363962 20079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:47.363989 20082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:47.363973 20080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:47.364264 20074 server_base.cc:1034] running on GCE node
I20260928 14:44:47.364698 20074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:47.365309 20074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:47.366488 20074 hybrid_clock.cc:648] HybridClock initialized: now 1790606687366477 us; error 35 us; skew 500 ppm
Sep 28 14:44:47 dist-test-slave-0n27 krb5kdc[19967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606687, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:47.369113 20074 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:44:47.369999 20074 webserver.cc:460] Webserver started at http://127.1.201.1:36507/ using document root <none> and password file <none>
I20260928 14:44:47.370540 20074 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:47.370603 20074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:47.370805 20074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:47.372689 20074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "e26e25d1974245528938142a221d0240"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "89f8c2f2a912b4d52268209be12b22e9"
server_key_iv: "39e458e5f5334d06933e872c19160bbc"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.373198 20074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "e26e25d1974245528938142a221d0240"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "89f8c2f2a912b4d52268209be12b22e9"
server_key_iv: "39e458e5f5334d06933e872c19160bbc"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.376621 20074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:44:47.378919 20089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:47.380060 20074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:44:47.380209 20074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "e26e25d1974245528938142a221d0240"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "89f8c2f2a912b4d52268209be12b22e9"
server_key_iv: "39e458e5f5334d06933e872c19160bbc"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.380314 20074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:47.404721 20074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:47.407414 20074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:47.407606 20074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:47.408262 20074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:47.409049 20074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:47.409114 20074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:47.409188 20074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:47.409235 20074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:47.417968 20074 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:35849
I20260928 14:44:47.417985 20202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:35849 every 8 connection(s)
I20260928 14:44:47.418803 20074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:44:47 dist-test-slave-0n27 krb5kdc[19967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606687, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:44:47.429009 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 20074
I20260928 14:44:47.429173 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:44:47.429502 1828 external_mini_cluster.cc:1428] Setting key a3d2e8d883389eff08420ab1cb0108c3
I20260928 14:44:47.432530 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.420535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:56043 (local address 127.1.201.62:34511)
0928 14:44:47.420662 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:47.420666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:47.421454 (+ 788us) server_negotiation.cc:407] Connection header received
0928 14:44:47.422769 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:47.422772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:47.422841 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:47.422943 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:47.424147 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.424689 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.425848 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.426085 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.426197 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.428636 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:47.428639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:47.428641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:47.428667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:47.430375 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:47.430951 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:47.430953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:47.430955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:47.431049 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:47.431373 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:47.431377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:47.431378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:47.431606 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:47.431707 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:47.432146 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 14:44:47.432288 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:44:47.433161 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.420894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34511 (local address 127.1.201.1:56043)
0928 14:44:47.421306 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:47.421347 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:47.422555 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:47.423115 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:47.423122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:47.423467 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:47.424001 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.424016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.424816 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.424821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.425687 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.425695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.426247 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.426251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.426422 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.427012 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:47.427028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:47.428486 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:47.430481 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:47.430486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:47.430500 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:47.430840 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:47.431149 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:47.431151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:47.431152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:47.431279 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:47.431712 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:47.431717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:47.431926 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 14:44:47.432070 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 14:44:47.432299 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":111,"threads_started":1}
I20260928 14:44:47.434342 20203 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34511
I20260928 14:44:47.434572 20203 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:47.435113 20203 heartbeater.cc:511] Master 127.1.201.62:34511 requested a full tablet report, sending...
I20260928 14:44:47.436616 20012 ts_manager.cc:194] Registered new tserver with Master: e26e25d1974245528938142a221d0240 (127.1.201.1:35849)
I20260928 14:44:47.437702 20012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:56043
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:44:47.456382 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34511
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:47.566227 20208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:47.566488 20208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:47.566558 20208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:47.570647 20208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:47.570719 20208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:47.570798 20208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:44:47.576493 20208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:47.577505 20208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:47.578574 20208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:47.587957 20214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:47.588008 20213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:47.588193 20216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:47.588189 20208 server_base.cc:1034] running on GCE node
I20260928 14:44:47.588654 20208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:47.589249 20208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:47.590428 20208 hybrid_clock.cc:648] HybridClock initialized: now 1790606687590398 us; error 45 us; skew 500 ppm
Sep 28 14:44:47 dist-test-slave-0n27 krb5kdc[19967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606687, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:47.592965 20208 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:44:47.593902 20208 webserver.cc:460] Webserver started at http://127.1.201.2:35693/ using document root <none> and password file <none>
I20260928 14:44:47.594444 20208 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:47.594532 20208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:47.594749 20208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:47.596670 20208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "ccbd89df35794128ae22677f5ee2f233"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "ef94381089af0f1e23c6c75155cbd58e"
server_key_iv: "5bbc8fdee8b2b0defbeaaa3270bd1457"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.597196 20208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "ccbd89df35794128ae22677f5ee2f233"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "ef94381089af0f1e23c6c75155cbd58e"
server_key_iv: "5bbc8fdee8b2b0defbeaaa3270bd1457"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.600739 20208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260928 14:44:47.603150 20223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:47.604300 20208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:44:47.604424 20208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "ccbd89df35794128ae22677f5ee2f233"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "ef94381089af0f1e23c6c75155cbd58e"
server_key_iv: "5bbc8fdee8b2b0defbeaaa3270bd1457"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.604557 20208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:47.630296 20208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:47.633159 20208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:47.633365 20208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:47.633957 20208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:47.634737 20208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:47.634809 20208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:47.634878 20208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:47.634927 20208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:47.643770 20208 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:40149
I20260928 14:44:47.643798 20336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:40149 every 8 connection(s)
I20260928 14:44:47.644631 20208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:44:47.652601 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 20208
I20260928 14:44:47.652751 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:44:47.653066 1828 external_mini_cluster.cc:1428] Setting key c5be123aa385253409eced7b7fe1ffa4
Sep 28 14:44:47 dist-test-slave-0n27 krb5kdc[19967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606687, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
I20260928 14:44:47.663609 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.646405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:50197 (local address 127.1.201.62:34511)
0928 14:44:47.646543 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:47.646547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:47.647348 (+ 801us) server_negotiation.cc:407] Connection header received
0928 14:44:47.648648 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:47.648652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:47.648710 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:47.648824 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:47.649953 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.650501 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.651656 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.651892 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.652004 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.660160 (+ 8156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:47.660163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:47.660165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:47.660190 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:47.661567 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:47.662160 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:47.662162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:47.662163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:47.662234 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:47.662623 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:47.662625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:47.662626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:47.662802 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:47.662876 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:47.663393 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 14:44:47.663494 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:44:47.664386 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.646772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34511 (local address 127.1.201.2:50197)
0928 14:44:47.647192 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:47.647232 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:47.648435 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:47.648989 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:47.648996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:47.649338 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:47.649805 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.649821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.650626 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.650636 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.651529 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.651538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.652041 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.652045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.652215 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.658134 (+ 5919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:47.658150 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:47.659967 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:47.661690 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:47.661698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:47.661714 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:47.662069 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:47.662330 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:47.662336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:47.662338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:47.662521 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:47.662893 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:47.662898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:47.663116 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 14:44:47.663307 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 14:44:47.663551 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
I20260928 14:44:47.665555 20337 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34511
I20260928 14:44:47.665855 20337 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:47.666422 20337 heartbeater.cc:511] Master 127.1.201.62:34511 requested a full tablet report, sending...
I20260928 14:44:47.667510 20012 ts_manager.cc:194] Registered new tserver with Master: ccbd89df35794128ae22677f5ee2f233 (127.1.201.2:40149)
I20260928 14:44:47.668074 20012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:50197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:44:47.679865 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34511
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:47.790381 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:47.790614 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:47.790688 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:47.794726 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:47.794775 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:47.794905 20342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:44:47.800055 20342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:47.800987 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:47.801950 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:47.811487 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:47.811537 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:47.811604 20342 server_base.cc:1034] running on GCE node
W20260928 14:44:47.811559 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:47.812170 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:47.812767 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:47.813943 20342 hybrid_clock.cc:648] HybridClock initialized: now 1790606687813912 us; error 44 us; skew 500 ppm
Sep 28 14:44:47 dist-test-slave-0n27 krb5kdc[19967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606687, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:47.816452 20342 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:44:47.817271 20342 webserver.cc:460] Webserver started at http://127.1.201.3:42075/ using document root <none> and password file <none>
I20260928 14:44:47.817857 20342 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:47.817965 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:47.818200 20342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:47.820219 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "57707f7d29144056939c33db287be7e6"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "af72694102aa2a5479451e89144026fe"
server_key_iv: "53dfd0d4fe09b0f570eec2761436a538"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.820770 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "57707f7d29144056939c33db287be7e6"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "af72694102aa2a5479451e89144026fe"
server_key_iv: "53dfd0d4fe09b0f570eec2761436a538"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.824241 20342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:44:47.826527 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:47.827664 20342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:47.827792 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "57707f7d29144056939c33db287be7e6"
format_stamp: "Formatted at 2026-09-28 14:44:47 on dist-test-slave-0n27"
server_key: "af72694102aa2a5479451e89144026fe"
server_key_iv: "53dfd0d4fe09b0f570eec2761436a538"
server_key_version: "encryptionkey@0"
I20260928 14:44:47.827914 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:47.844954 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:47.847867 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:47.848080 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:47.848647 20342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:47.849443 20342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:47.849510 20342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:47.849582 20342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:47.849625 20342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:47.858357 20342 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:39559
I20260928 14:44:47.858394 20470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:39559 every 8 connection(s)
I20260928 14:44:47.859221 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:44:47.866418 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 20342
I20260928 14:44:47.866557 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:44:47.866868 1828 external_mini_cluster.cc:1428] Setting key 8558436b2880007e536f34a33e6a0cd4
Sep 28 14:44:47 dist-test-slave-0n27 krb5kdc[19967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606687, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:44:47.872241 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.860861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:51803 (local address 127.1.201.62:34511)
0928 14:44:47.860997 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:47.861001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:47.861712 (+ 711us) server_negotiation.cc:407] Connection header received
0928 14:44:47.862979 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:47.862983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:47.863042 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:47.863139 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:47.864248 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.864763 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.865887 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.866097 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.866230 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.869023 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:47.869027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:47.869029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:47.869055 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:47.870569 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:47.871038 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:47.871041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:47.871042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:47.871112 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:47.871394 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:47.871396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:47.871397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:47.871554 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:47.871625 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:47.872041 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 14:44:47.872129 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:44:47.873035 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.861155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:34511 (local address 127.1.201.3:51803)
0928 14:44:47.861563 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:47.861602 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:47.862776 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:47.863306 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:47.863312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:47.863643 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:47.864097 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.864110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.864884 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.864888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.865705 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.865722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.866240 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.866243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.866435 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.867125 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:47.867145 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:47.868839 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:47.870708 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:47.870713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:47.870724 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:47.870958 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:47.871199 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:47.871201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:47.871202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:47.871320 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:47.871635 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:47.871640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:47.871848 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 14:44:47.871999 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 14:44:47.872225 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":114,"threads_started":1}
I20260928 14:44:47.874143 20471 heartbeater.cc:347] Connected to a master server at 127.1.201.62:34511
I20260928 14:44:47.874366 20471 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:47.874826 20471 heartbeater.cc:511] Master 127.1.201.62:34511 requested a full tablet report, sending...
I20260928 14:44:47.875795 20012 ts_manager.cc:194] Registered new tserver with Master: 57707f7d29144056939c33db287be7e6 (127.1.201.3:39559)
I20260928 14:44:47.876336 20012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:51803
I20260928 14:44:47.880682 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:44:47.889674 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.882641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.1.201.62:34511)
0928 14:44:47.882783 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:47.882787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:47.883034 (+ 247us) server_negotiation.cc:407] Connection header received
0928 14:44:47.883124 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:47.883126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:47.883167 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:47.883228 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:47.883985 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.884459 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.885309 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.885504 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.885578 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.886975 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:47.886978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:47.886980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:47.887000 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:47.888178 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:47.888626 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:47.888628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:47.888629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:47.888699 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:47.888960 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:47.888962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:47.888963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:47.889155 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:47.889271 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:47.889479 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 14:44:47.889564 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 14:44:47.892309 20012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:44:47.895339 20012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:44:47.908676 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.902802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:39559 (local address 127.0.0.1:44972)
0928 14:44:47.903632 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:47.903682 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:47.903808 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:47.904562 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:47.904567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:47.904589 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:47.904779 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.904785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.906219 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.906221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.906953 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.906961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.908125 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.908128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.908240 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.908277 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:44:47.908395 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 14:44:47.908510 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":38,"spinlock_wait_cycles":6144,"thread_start_us":72,"threads_started":1}
I20260928 14:44:47.908936 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.903445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.1.201.1:35849)
0928 14:44:47.903839 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:47.903854 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:47.903879 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:44:47.904119 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:47.904125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:47.904335 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:47.904500 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:47.905050 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.905744 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.907274 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.908121 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.908228 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.908373 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:47.908693 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 14:44:47.908785 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":108,"threads_started":1}
I20260928 14:44:47.908885 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.903191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:40149 (local address 127.0.0.1:40538)
0928 14:44:47.904011 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:47.904022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:47.904089 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:47.904691 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:47.904694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:47.904705 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:47.904832 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.904841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.906180 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.906183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.907294 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.907302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.908403 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.908407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.908558 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.908607 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 14:44:47.908681 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:44:47.908754 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":35,"thread_start_us":85,"threads_started":1}
I20260928 14:44:47.909037 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.902975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35849 (local address 127.0.0.1:47772)
0928 14:44:47.903632 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:47.903682 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:47.903808 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:47.904654 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:47.904658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:47.904696 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:47.904944 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.904949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.905882 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.905886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.907137 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:47.907147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.908399 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.908403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:47.908498 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.908517 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:44:47.908643 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:44:47.908732 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":56,"spinlock_wait_cycles":4096,"thread_start_us":85,"threads_started":1}
I20260928 14:44:47.909621 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.903593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.1.201.2:40149)
0928 14:44:47.904316 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:47.904326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:47.904345 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:44:47.904413 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:47.904418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:47.904577 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:47.904722 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:47.904937 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.906056 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.907454 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.908269 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.909235 (+ 966us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.909368 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:47.909422 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 14:44:47.909510 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":141,"threads_started":1}
I20260928 14:44:47.910384 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:47.902961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44972 (local address 127.1.201.3:39559)
0928 14:44:47.904077 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:47.904087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:47.904110 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:44:47.904197 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:47.904203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:47.904393 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:47.904782 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:47.904937 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.906048 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.907072 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:47.908003 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:47.908126 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:47.908327 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:47.910150 (+ 1823us) server_negotiation.cc:299] Negotiation successful
0928 14:44:47.910225 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":125,"threads_started":1}
I20260928 14:44:47.912123 20271 tablet_service.cc:1467] Processing CreateTablet for tablet eb80109441524700a4f72b8e47725972 (DEFAULT_TABLE table=test-table [id=f32fdaa65f9b46bfb2ecd026c3b562e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:47.912154 20137 tablet_service.cc:1467] Processing CreateTablet for tablet eb80109441524700a4f72b8e47725972 (DEFAULT_TABLE table=test-table [id=f32fdaa65f9b46bfb2ecd026c3b562e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:47.913022 20271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb80109441524700a4f72b8e47725972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:47.912727 20405 tablet_service.cc:1467] Processing CreateTablet for tablet eb80109441524700a4f72b8e47725972 (DEFAULT_TABLE table=test-table [id=f32fdaa65f9b46bfb2ecd026c3b562e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:47.913108 20137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb80109441524700a4f72b8e47725972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:47.913504 20405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb80109441524700a4f72b8e47725972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:47.918582 20488 tablet_bootstrap.cc:492] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: Bootstrap starting.
I20260928 14:44:47.920308 20489 tablet_bootstrap.cc:492] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240: Bootstrap starting.
I20260928 14:44:47.921641 20488 tablet_bootstrap.cc:654] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:47.922035 20489 tablet_bootstrap.cc:654] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:47.922117 20490 tablet_bootstrap.cc:492] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: Bootstrap starting.
I20260928 14:44:47.922479 20488 log.cc:824] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:47.923254 20489 log.cc:824] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:47.924177 20490 tablet_bootstrap.cc:654] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:47.924383 20488 tablet_bootstrap.cc:492] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: No bootstrap required, opened a new log
I20260928 14:44:47.924572 20488 ts_tablet_manager.cc:1397] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:44:47.925246 20489 tablet_bootstrap.cc:492] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240: No bootstrap required, opened a new log
I20260928 14:44:47.925316 20490 log.cc:824] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:47.925442 20489 ts_tablet_manager.cc:1397] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:44:47.927148 20490 tablet_bootstrap.cc:492] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: No bootstrap required, opened a new log
I20260928 14:44:47.927341 20490 ts_tablet_manager.cc:1397] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:44:47.927779 20488 raft_consensus.cc:348] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57707f7d29144056939c33db287be7e6" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39559 } } peers { permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 } } peers { permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 } }
I20260928 14:44:47.928035 20488 raft_consensus.cc:374] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:47.928088 20488 raft_consensus.cc:729] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbd89df35794128ae22677f5ee2f233, State: Initialized, Role: FOLLOWER
I20260928 14:44:47.928519 20488 consensus_queue.cc:260] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57707f7d29144056939c33db287be7e6" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39559 } } peers { permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 } } peers { permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 } }
I20260928 14:44:47.929160 20489 raft_consensus.cc:348] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57707f7d29144056939c33db287be7e6" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39559 } } peers { permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 } } peers { permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 } }
I20260928 14:44:47.929334 20488 ts_tablet_manager.cc:1428] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:44:47.929396 20489 raft_consensus.cc:374] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:47.929448 20489 raft_consensus.cc:729] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26e25d1974245528938142a221d0240, State: Initialized, Role: FOLLOWER
I20260928 14:44:47.929704 20337 heartbeater.cc:503] Master 127.1.201.62:34511 was elected leader, sending a full tablet report...
I20260928 14:44:47.929890 20489 consensus_queue.cc:260] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57707f7d29144056939c33db287be7e6" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39559 } } peers { permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 } } peers { permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 } }
I20260928 14:44:47.930509 20203 heartbeater.cc:503] Master 127.1.201.62:34511 was elected leader, sending a full tablet report...
I20260928 14:44:47.930858 20489 ts_tablet_manager.cc:1428] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:44:47.931021 20490 raft_consensus.cc:348] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57707f7d29144056939c33db287be7e6" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39559 } } peers { permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 } } peers { permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 } }
I20260928 14:44:47.931262 20490 raft_consensus.cc:374] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:47.931298 20490 raft_consensus.cc:729] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57707f7d29144056939c33db287be7e6, State: Initialized, Role: FOLLOWER
I20260928 14:44:47.931730 20490 consensus_queue.cc:260] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57707f7d29144056939c33db287be7e6" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39559 } } peers { permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 } } peers { permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 } }
I20260928 14:44:47.932607 20490 ts_tablet_manager.cc:1428] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:44:47.932687 20471 heartbeater.cc:503] Master 127.1.201.62:34511 was elected leader, sending a full tablet report...
I20260928 14:44:48.079437 20496 raft_consensus.cc:482] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:48.079689 20496 raft_consensus.cc:504] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57707f7d29144056939c33db287be7e6" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39559 } } peers { permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 } } peers { permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 } }
I20260928 14:44:48.080817 20496 leader_election.cc:288] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e26e25d1974245528938142a221d0240 (127.1.201.1:35849), ccbd89df35794128ae22677f5ee2f233 (127.1.201.2:40149)
I20260928 14:44:48.084905 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.081108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:40149 (local address 127.1.201.3:39035)
0928 14:44:48.081324 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:48.081340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:48.081460 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:48.081774 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:48.081795 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:48.081818 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:48.081935 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:48.081943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.082792 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.082795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:48.083828 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:48.083838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.084462 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.084468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:48.084593 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:48.084617 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:44:48.084687 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:44:48.084755 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":51}
I20260928 14:44:48.085065 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.081203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:41981 (local address 127.1.201.1:35849)
0928 14:44:48.081333 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:48.081338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:48.081702 (+ 364us) server_negotiation.cc:407] Connection header received
0928 14:44:48.081867 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:48.081870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:48.081922 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:48.082005 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:48.082282 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.082781 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.084023 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.084446 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.084562 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:48.084641 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:48.084869 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 14:44:48.084950 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:44:48.085099 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.081108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35849 (local address 127.1.201.3:41981)
0928 14:44:48.081583 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:48.081597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:48.081729 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:48.082054 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:48.082056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:48.082074 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:48.082178 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:48.082184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.082877 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.082880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:48.083848 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:48.083857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.084595 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.084601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:48.084729 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:48.084761 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 14:44:48.084836 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:44:48.084898 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":48,"thread_start_us":125,"threads_started":1}
I20260928 14:44:48.085187 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.081216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:39035 (local address 127.1.201.2:40149)
0928 14:44:48.081363 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:48.081368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:48.081386 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:44:48.081598 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:48.081604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:48.081666 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:48.081747 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:48.082046 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.082680 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.083946 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.084351 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.084475 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:48.084580 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:48.084997 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 14:44:48.085070 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:44:48.085769 20157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb80109441524700a4f72b8e47725972" candidate_uuid: "57707f7d29144056939c33db287be7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26e25d1974245528938142a221d0240" is_pre_election: true
I20260928 14:44:48.085837 20291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb80109441524700a4f72b8e47725972" candidate_uuid: "57707f7d29144056939c33db287be7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbd89df35794128ae22677f5ee2f233" is_pre_election: true
I20260928 14:44:48.086169 20291 raft_consensus.cc:2454] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57707f7d29144056939c33db287be7e6 in term 0.
I20260928 14:44:48.086175 20157 raft_consensus.cc:2454] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57707f7d29144056939c33db287be7e6 in term 0.
I20260928 14:44:48.086651 20358 leader_election.cc:302] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57707f7d29144056939c33db287be7e6, e26e25d1974245528938142a221d0240; no voters:
I20260928 14:44:48.086938 20496 raft_consensus.cc:2781] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:44:48.087019 20496 raft_consensus.cc:482] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:44:48.087060 20496 raft_consensus.cc:3037] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:48.088071 20496 raft_consensus.cc:504] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57707f7d29144056939c33db287be7e6" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39559 } } peers { permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 } } peers { permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 } }
I20260928 14:44:48.088486 20496 leader_election.cc:288] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [CANDIDATE]: Term 1 election: Requested vote from peers e26e25d1974245528938142a221d0240 (127.1.201.1:35849), ccbd89df35794128ae22677f5ee2f233 (127.1.201.2:40149)
I20260928 14:44:48.088917 20291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb80109441524700a4f72b8e47725972" candidate_uuid: "57707f7d29144056939c33db287be7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbd89df35794128ae22677f5ee2f233"
I20260928 14:44:48.088955 20157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb80109441524700a4f72b8e47725972" candidate_uuid: "57707f7d29144056939c33db287be7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26e25d1974245528938142a221d0240"
I20260928 14:44:48.089069 20157 raft_consensus.cc:3037] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:48.089069 20291 raft_consensus.cc:3037] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:48.090174 20157 raft_consensus.cc:2454] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57707f7d29144056939c33db287be7e6 in term 1.
I20260928 14:44:48.090231 20291 raft_consensus.cc:2454] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57707f7d29144056939c33db287be7e6 in term 1.
I20260928 14:44:48.090457 20358 leader_election.cc:302] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57707f7d29144056939c33db287be7e6, e26e25d1974245528938142a221d0240; no voters:
I20260928 14:44:48.090624 20496 raft_consensus.cc:2781] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:48.090821 20496 raft_consensus.cc:686] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 1 LEADER]: Becoming Leader. State: Replica: 57707f7d29144056939c33db287be7e6, State: Running, Role: LEADER
I20260928 14:44:48.091157 20496 consensus_queue.cc:237] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57707f7d29144056939c33db287be7e6" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39559 } } peers { permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 } } peers { permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 } }
I20260928 14:44:48.094349 20011 catalog_manager.cc:5495] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57707f7d29144056939c33db287be7e6 (127.1.201.3). New cstate: current_term: 1 leader_uuid: "57707f7d29144056939c33db287be7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57707f7d29144056939c33db287be7e6" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 } health_report { overall_health: UNKNOWN } } }
W20260928 14:44:48.110077 20472 tablet.cc:2369] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:48.143275 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.139556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.1.201.3:39559)
0928 14:44:48.139709 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:48.139713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:48.139728 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:44:48.139849 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:48.139852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:48.139906 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:48.140031 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:44:48.140288 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.140778 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.141611 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.141894 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.142000 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:48.142304 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:44:48.142822 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:44:48.142977 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:48.143073 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 14:44:48.143113 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 14:44:48.145761 20338 tablet.cc:2369] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:48.149202 20157 raft_consensus.cc:1261] T eb80109441524700a4f72b8e47725972 P e26e25d1974245528938142a221d0240 [term 1 FOLLOWER]: Refusing update from remote peer 57707f7d29144056939c33db287be7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:48.149257 20291 raft_consensus.cc:1261] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233 [term 1 FOLLOWER]: Refusing update from remote peer 57707f7d29144056939c33db287be7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:48.149972 20498 consensus_queue.cc:1035] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccbd89df35794128ae22677f5ee2f233" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:48.150162 20496 consensus_queue.cc:1035] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26e25d1974245528938142a221d0240" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:48.164788 20503 mvcc.cc:205] Tried to move back new op lower bound from 7334324994651439104 to 7334324994424082432. Current Snapshot: MvccSnapshot[applied={T|T < 7334324994651439104}]
I20260928 14:44:48.166836 20012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:48.166961 20012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:48.169116 20012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1a57007cf7143ad8dc9f4d5fc1c7cf0: Sending DeleteTablet for 3 replicas of tablet eb80109441524700a4f72b8e47725972
I20260928 14:44:48.169829 20405 tablet_service.cc:1514] Processing DeleteTablet for tablet eb80109441524700a4f72b8e47725972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:48 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:44972
I20260928 14:44:48.169970 20137 tablet_service.cc:1514] Processing DeleteTablet for tablet eb80109441524700a4f72b8e47725972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:48 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:47772
I20260928 14:44:48.170545 20271 tablet_service.cc:1514] Processing DeleteTablet for tablet eb80109441524700a4f72b8e47725972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:48 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:40538
I20260928 14:44:48.170903 20513 tablet_replica.cc:330] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: stopping tablet replica
I20260928 14:44:48.170993 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 20074
I20260928 14:44:48.171270 20513 raft_consensus.cc:2229] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:48.171622 20513 raft_consensus.cc:2258] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:48.172837 20513 ts_tablet_manager.cc:1905] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:48.174681 20511 tablet_replica.cc:330] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: stopping tablet replica
I20260928 14:44:48.174934 20511 raft_consensus.cc:2229] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:44:48.175290 20511 raft_consensus.cc:2258] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:48.176035 20513 ts_tablet_manager.cc:1918] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:48.176122 20513 log.cc:1196] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/eb80109441524700a4f72b8e47725972
I20260928 14:44:48.176437 20513 ts_tablet_manager.cc:1939] T eb80109441524700a4f72b8e47725972 P ccbd89df35794128ae22677f5ee2f233: Deleting consensus metadata
I20260928 14:44:48.177557 20511 ts_tablet_manager.cc:1905] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:48.177546 19999 catalog_manager.cc:4841] TS ccbd89df35794128ae22677f5ee2f233 (127.1.201.2:40149): tablet eb80109441524700a4f72b8e47725972 (table test-table [id=f32fdaa65f9b46bfb2ecd026c3b562e2]) successfully deleted
I20260928 14:44:48.179589 20511 ts_tablet_manager.cc:1918] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:48.179697 20511 log.cc:1196] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/eb80109441524700a4f72b8e47725972
I20260928 14:44:48.179989 20511 ts_tablet_manager.cc:1939] T eb80109441524700a4f72b8e47725972 P 57707f7d29144056939c33db287be7e6: Deleting consensus metadata
W20260928 14:44:48.181070 19996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:35849 (error 108)
I20260928 14:44:48.181237 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 20208
I20260928 14:44:48.181088 19997 catalog_manager.cc:4841] TS 57707f7d29144056939c33db287be7e6 (127.1.201.3:39559): tablet eb80109441524700a4f72b8e47725972 (table test-table [id=f32fdaa65f9b46bfb2ecd026c3b562e2]) successfully deleted
I20260928 14:44:48.183051 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.182719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:35849 (local address 127.0.0.1:47774)
0928 14:44:48.182871 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:48.182953 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:35849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:44:48.183236 19996 catalog_manager.cc:4584] TS e26e25d1974245528938142a221d0240 (127.1.201.1:35849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb80109441524700a4f72b8e47725972: Network error: Client connection negotiation failed: client connection to 127.1.201.1:35849: connect: Connection refused (error 111)
I20260928 14:44:48.189087 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 20342
I20260928 14:44:48.195044 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 19980
2026-09-28T14:44:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2377 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T14:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:48Z Disabled control of system clock
I20260928 14:44:48.225049 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:41817
--webserver_interface=127.1.201.62
--webserver_port=0
--builtin_ntp_servers=127.1.201.20:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:41817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:44:48.331044 20518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:48.331307 20518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:48.331342 20518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:48.335160 20518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:48.335206 20518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:48.335220 20518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:48.335237 20518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:48.335250 20518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:48.339883 20518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:41817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:41817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:48.340864 20518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:48.342226 20518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:48.350451 20526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:48.350445 20523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:48.350445 20524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:48.350878 20518 server_base.cc:1034] running on GCE node
I20260928 14:44:48.351394 20518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:48.352442 20518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:48.353622 20518 hybrid_clock.cc:648] HybridClock initialized: now 1790606688353604 us; error 30 us; skew 500 ppm
I20260928 14:44:48.355345 20518 webserver.cc:460] Webserver started at http://127.1.201.62:43727/ using document root <none> and password file <none>
I20260928 14:44:48.355906 20518 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:48.355995 20518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:48.356220 20518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:48.358283 20518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "7f046e8f3e21494d8913744249e61185"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "f4eabdff5c17685f1875c187628a9852"
server_key_iv: "429882759d50f74e493d7af15328690c"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.358829 20518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "7f046e8f3e21494d8913744249e61185"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "f4eabdff5c17685f1875c187628a9852"
server_key_iv: "429882759d50f74e493d7af15328690c"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.362277 20518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:44:48.364521 20532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:48.365695 20518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:48.365842 20518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "7f046e8f3e21494d8913744249e61185"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "f4eabdff5c17685f1875c187628a9852"
server_key_iv: "429882759d50f74e493d7af15328690c"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.365949 20518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:48.376322 20518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:48.376901 20518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:48.377094 20518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:48.383721 20518 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:41817
I20260928 14:44:48.383728 20584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:41817 every 8 connection(s)
I20260928 14:44:48.384753 20518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:48.387779 20585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:48.390234 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 20518
I20260928 14:44:48.390354 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:48.390688 1828 external_mini_cluster.cc:1428] Setting key dec097d5763d4275325febad48a0b278
I20260928 14:44:48.393736 20585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185: Bootstrap starting.
I20260928 14:44:48.395943 20585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:48.396634 20585 log.cc:824] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:48.398558 20585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185: No bootstrap required, opened a new log
I20260928 14:44:48.399603 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.392115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.1.201.62:41817)
0928 14:44:48.392717 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:48.392731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:48.392763 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:44:48.393347 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:48.393362 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:48.393748 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:48.394138 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:48.394561 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.395492 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.396449 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.396963 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.397127 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:48.398010 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:48.398014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:48.398026 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:48.398388 (+ 362us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:48.398439 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:48.398459 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:48.398566 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:48.398703 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 14:44:48.398947 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
I20260928 14:44:48.402626 20585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f046e8f3e21494d8913744249e61185" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41817 } }
I20260928 14:44:48.402875 20585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:48.402971 20585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f046e8f3e21494d8913744249e61185, State: Initialized, Role: FOLLOWER
I20260928 14:44:48.403385 20585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f046e8f3e21494d8913744249e61185" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41817 } }
I20260928 14:44:48.403544 20585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:48.403635 20585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:48.403730 20585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:48.404556 20585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f046e8f3e21494d8913744249e61185" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41817 } }
I20260928 14:44:48.404879 20585 leader_election.cc:302] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f046e8f3e21494d8913744249e61185; no voters:
I20260928 14:44:48.405174 20585 leader_election.cc:288] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:48.405285 20590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:48.405511 20590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [term 1 LEADER]: Becoming Leader. State: Replica: 7f046e8f3e21494d8913744249e61185, State: Running, Role: LEADER
I20260928 14:44:48.405850 20590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f046e8f3e21494d8913744249e61185" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41817 } }
I20260928 14:44:48.406414 20585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:48.407429 20591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f046e8f3e21494d8913744249e61185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f046e8f3e21494d8913744249e61185" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41817 } } }
I20260928 14:44:48.407553 20591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:48.407418 20592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f046e8f3e21494d8913744249e61185. Latest consensus state: current_term: 1 leader_uuid: "7f046e8f3e21494d8913744249e61185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f046e8f3e21494d8913744249e61185" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41817 } } }
I20260928 14:44:48.407829 20592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185 [sys.catalog]: This master's current role is: LEADER
W20260928 14:44:48.411150 20605 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:44:48.411216 20605 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:44:48.411290 20599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:48.411959 20599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:48.417366 20599 catalog_manager.cc:1347] Generated new cluster ID: e963f19292134161b9275a1bb0d1c769
I20260928 14:44:48.417430 20599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:48.450162 20599 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:48.451350 20599 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:48.465389 20599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185: Generated new TSK 0
I20260928 14:44:48.466166 20599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:44:48.476356 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:41817
--builtin_ntp_servers=127.1.201.20:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:44:48.586431 20609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:48.586673 20609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:48.586717 20609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:48.590579 20609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:48.590624 20609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:48.590675 20609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:44:48.595170 20609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:41817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:48.596108 20609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:48.597384 20609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:48.606681 20617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:48.606731 20615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:48.606688 20614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:48.606885 20609 server_base.cc:1034] running on GCE node
I20260928 14:44:48.607347 20609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:48.608035 20609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:48.609222 20609 hybrid_clock.cc:648] HybridClock initialized: now 1790606688609209 us; error 34 us; skew 500 ppm
I20260928 14:44:48.611130 20609 webserver.cc:460] Webserver started at http://127.1.201.1:46105/ using document root <none> and password file <none>
I20260928 14:44:48.611697 20609 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:48.611821 20609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:48.612043 20609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:48.614085 20609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "bce815d4cddf4407a9b0b4bb495365b9"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "c27d7aa310f735242d96f4674ff4132f"
server_key_iv: "bb192da6b720ae423b3fb4f8fb207e66"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.614629 20609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "bce815d4cddf4407a9b0b4bb495365b9"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "c27d7aa310f735242d96f4674ff4132f"
server_key_iv: "bb192da6b720ae423b3fb4f8fb207e66"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.618363 20609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:44:48.620711 20623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:48.621975 20609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:48.622098 20609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "bce815d4cddf4407a9b0b4bb495365b9"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "c27d7aa310f735242d96f4674ff4132f"
server_key_iv: "bb192da6b720ae423b3fb4f8fb207e66"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.622205 20609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:48.635393 20609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:48.635994 20609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:48.636178 20609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:48.636711 20609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:48.637593 20609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:48.637663 20609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:48.637739 20609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:48.637809 20609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:48.646773 20609 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:44663
I20260928 14:44:48.646808 20736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:44663 every 8 connection(s)
I20260928 14:44:48.647600 20609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:44:48.652662 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 20609
I20260928 14:44:48.652809 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:44:48.653126 1828 external_mini_cluster.cc:1428] Setting key e85750893add1f0e07bcde4d65de3905
I20260928 14:44:48.655267 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:41817
--builtin_ntp_servers=127.1.201.20:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:44:48.657824 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.649380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:46073 (local address 127.1.201.62:41817)
0928 14:44:48.649516 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:48.649519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:48.650596 (+ 1077us) server_negotiation.cc:407] Connection header received
0928 14:44:48.652032 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:48.652036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:48.652097 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:48.652199 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:48.653359 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.654096 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.655720 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.655933 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.656009 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:48.657270 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:48.657277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:48.657282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:48.657298 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:48.657324 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:48.657336 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:48.657414 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:48.657555 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 14:44:48.657626 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:44:48.658582 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.649746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:41817 (local address 127.1.201.1:46073)
0928 14:44:48.650438 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:48.650476 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:48.651810 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:48.652393 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:48.652419 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:48.652526 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:48.653189 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:48.653208 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.654225 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.654228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:48.655563 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:48.655593 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.656086 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.656092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:48.656359 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:48.656992 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:44:48.656993 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:48.657031 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:44:48.657092 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:48.657420 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:48.657427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:48.657431 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:44:48.657520 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:44:48.657723 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260928 14:44:48.659765 20737 heartbeater.cc:347] Connected to a master server at 127.1.201.62:41817
I20260928 14:44:48.659976 20737 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:48.660454 20737 heartbeater.cc:511] Master 127.1.201.62:41817 requested a full tablet report, sending...
I20260928 14:44:48.661927 20549 ts_manager.cc:194] Registered new tserver with Master: bce815d4cddf4407a9b0b4bb495365b9 (127.1.201.1:44663)
I20260928 14:44:48.663046 20549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.1:46073
W20260928 14:44:48.764953 20740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:48.765197 20740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:48.765233 20740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:48.769269 20740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:48.769315 20740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:48.769412 20740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:44:48.774017 20740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:41817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:48.774976 20740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:48.776290 20740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:48.785408 20748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:48.785384 20746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:48.785396 20745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:48.785830 20740 server_base.cc:1034] running on GCE node
I20260928 14:44:48.786209 20740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:48.786775 20740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:48.788002 20740 hybrid_clock.cc:648] HybridClock initialized: now 1790606688787991 us; error 31 us; skew 500 ppm
I20260928 14:44:48.789839 20740 webserver.cc:460] Webserver started at http://127.1.201.2:35091/ using document root <none> and password file <none>
I20260928 14:44:48.790409 20740 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:48.790499 20740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:48.790709 20740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:48.792729 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "d4155064161a4552b246266e54f3b813"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "549ac8edf3b0904a021dcfdc62ccb52e"
server_key_iv: "6d38753e7e0ce60a67c2cc7f0b28be9f"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.793296 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "d4155064161a4552b246266e54f3b813"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "549ac8edf3b0904a021dcfdc62ccb52e"
server_key_iv: "6d38753e7e0ce60a67c2cc7f0b28be9f"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.796901 20740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:44:48.799293 20754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:48.800397 20740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:44:48.800530 20740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "d4155064161a4552b246266e54f3b813"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "549ac8edf3b0904a021dcfdc62ccb52e"
server_key_iv: "6d38753e7e0ce60a67c2cc7f0b28be9f"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.800642 20740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:48.816129 20740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:48.816740 20740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:48.816934 20740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:48.817469 20740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:48.818397 20740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:48.818470 20740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:48.818542 20740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:48.818585 20740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:48.826695 20740 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:40113
I20260928 14:44:48.826719 20867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:40113 every 8 connection(s)
I20260928 14:44:48.827508 20740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:44:48.832132 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 20740
I20260928 14:44:48.832274 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:44:48.832599 1828 external_mini_cluster.cc:1428] Setting key 7eb0e2c7d99aba602837e5f648e69f04
I20260928 14:44:48.834766 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:41817
--builtin_ntp_servers=127.1.201.20:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:44:48.837709 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.829231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:44359 (local address 127.1.201.62:41817)
0928 14:44:48.829391 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:48.829396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:48.830408 (+ 1012us) server_negotiation.cc:407] Connection header received
0928 14:44:48.831753 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:48.831756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:48.831802 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:48.831874 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:48.833091 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.833845 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.835526 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.835773 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.835855 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:48.837151 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:48.837157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:48.837160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:48.837176 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:48.837200 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:48.837211 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:48.837293 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:48.837434 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 14:44:48.837502 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":63}
I20260928 14:44:48.838495 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:48.829618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:41817 (local address 127.1.201.2:44359)
0928 14:44:48.830265 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:48.830303 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:48.831554 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:48.832115 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:48.832125 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:48.832237 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:48.832913 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:48.832932 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.833995 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.833998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:48.835398 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:48.835413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:48.835940 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:48.835946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:48.836176 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:48.836885 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:44:48.836886 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:48.836926 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:44:48.836987 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:48.837298 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:48.837305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:48.837309 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:44:48.837401 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:44:48.837616 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":152,"threads_started":1}
I20260928 14:44:48.839646 20868 heartbeater.cc:347] Connected to a master server at 127.1.201.62:41817
I20260928 14:44:48.839864 20868 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:48.840348 20868 heartbeater.cc:511] Master 127.1.201.62:41817 requested a full tablet report, sending...
I20260928 14:44:48.841346 20549 ts_manager.cc:194] Registered new tserver with Master: d4155064161a4552b246266e54f3b813 (127.1.201.2:40113)
I20260928 14:44:48.841955 20549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.2:44359
W20260928 14:44:48.942628 20871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:48.942888 20871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:48.942942 20871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:48.946935 20871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:48.946985 20871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:48.947036 20871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:44:48.951732 20871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:41817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:48.952754 20871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:48.954142 20871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:48.963425 20879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:48.963400 20876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:48.963465 20871 server_base.cc:1034] running on GCE node
W20260928 14:44:48.963474 20877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:48.964061 20871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:48.964679 20871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:48.965863 20871 hybrid_clock.cc:648] HybridClock initialized: now 1790606688965831 us; error 26 us; skew 500 ppm
I20260928 14:44:48.967628 20871 webserver.cc:460] Webserver started at http://127.1.201.3:41743/ using document root <none> and password file <none>
I20260928 14:44:48.968205 20871 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:48.968295 20871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:48.968523 20871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:48.970824 20871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "3ccc7e1fa24147178334a442c7dfb4fb"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "b6293b5ca611d0062d83ccef55ec5007"
server_key_iv: "02033c6a68a8ebe301364f71d1a92c89"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.971391 20871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "3ccc7e1fa24147178334a442c7dfb4fb"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "b6293b5ca611d0062d83ccef55ec5007"
server_key_iv: "02033c6a68a8ebe301364f71d1a92c89"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.974831 20871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:44:48.977077 20885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:48.978313 20871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:48.978425 20871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "3ccc7e1fa24147178334a442c7dfb4fb"
format_stamp: "Formatted at 2026-09-28 14:44:48 on dist-test-slave-0n27"
server_key: "b6293b5ca611d0062d83ccef55ec5007"
server_key_iv: "02033c6a68a8ebe301364f71d1a92c89"
server_key_version: "encryptionkey@0"
I20260928 14:44:48.978525 20871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:48.995430 20871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:48.996075 20871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:48.996266 20871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:48.996825 20871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:48.997756 20871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:48.997824 20871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:48.997925 20871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:48.997962 20871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:49.007210 20871 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:39981
I20260928 14:44:49.007248 20998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:39981 every 8 connection(s)
I20260928 14:44:49.008174 20871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:44:49.011158 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 20871
I20260928 14:44:49.011279 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:44:49.011626 1828 external_mini_cluster.cc:1428] Setting key 9c0311768c3bfa2c07a9e6c57fc67a2d
I20260928 14:44:49.017408 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.010009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:37015 (local address 127.1.201.62:41817)
0928 14:44:49.010149 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:49.010153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:49.011165 (+ 1012us) server_negotiation.cc:407] Connection header received
0928 14:44:49.012968 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:49.012971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:49.013036 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:49.013116 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:49.014103 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.014616 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.015711 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.015931 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.016031 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.016952 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:49.016955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:49.016958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:49.016975 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:49.016991 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:49.016998 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:49.017066 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:49.017230 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 14:44:49.017282 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:44:49.018250 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.010376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:41817 (local address 127.1.201.3:37015)
0928 14:44:49.011005 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:49.011056 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:49.012773 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:49.013301 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:49.013308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:49.013388 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:49.013972 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.013986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.014739 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.014742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.015567 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.015576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.016066 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.016069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.016230 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.016725 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:44:49.016726 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:49.016761 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:44:49.016817 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:49.017078 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:49.017087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:49.017093 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:44:49.017216 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:44:49.017461 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
I20260928 14:44:49.019296 20999 heartbeater.cc:347] Connected to a master server at 127.1.201.62:41817
I20260928 14:44:49.019498 20999 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:49.019968 20999 heartbeater.cc:511] Master 127.1.201.62:41817 requested a full tablet report, sending...
I20260928 14:44:49.020908 20549 ts_manager.cc:194] Registered new tserver with Master: 3ccc7e1fa24147178334a442c7dfb4fb (127.1.201.3:39981)
I20260928 14:44:49.021418 20549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.3:37015
I20260928 14:44:49.025250 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:44:49.030505 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.027155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.1.201.62:41817)
0928 14:44:49.027295 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:49.027298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:49.027450 (+ 152us) server_negotiation.cc:407] Connection header received
0928 14:44:49.027527 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:49.027528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:49.027566 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:49.027624 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:49.027829 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.028337 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.029103 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.029317 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.029387 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.030154 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:49.030157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:49.030159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:49.030169 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:49.030185 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:49.030193 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:49.030289 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:49.030384 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 14:44:49.030411 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:44:49.033191 20549 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:44:49.036267 20549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:44:49.050052 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.044857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.1.201.1:44663)
0928 14:44:49.045846 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:49.045860 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:49.045887 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:44:49.045964 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:49.045970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:49.046181 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:49.046320 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:49.046592 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.047409 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.048841 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.049371 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.049488 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.049682 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:49.049821 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 14:44:49.049913 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":138,"threads_started":1}
I20260928 14:44:49.050046 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.044711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:44663 (local address 127.0.0.1:40838)
0928 14:44:49.045352 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:49.045399 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:49.045513 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:49.046314 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:49.046319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:49.046337 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:49.046475 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.046481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.047518 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.047524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.048698 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.048709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.049491 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.049494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.049595 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.049641 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:44:49.049752 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:44:49.049896 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":31,"thread_start_us":76,"threads_started":1}
I20260928 14:44:49.051301 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.044840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:39981 (local address 127.0.0.1:36022)
0928 14:44:49.045722 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:49.045738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:49.045864 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:49.046635 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:49.046638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:49.046658 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:49.046768 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.046775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.048368 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.048371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.049816 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.049828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.050950 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.050954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.051034 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.051055 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:44:49.051125 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:44:49.051184 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":12,"thread_start_us":88,"threads_started":1}
I20260928 14:44:49.051939 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.045416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:40113 (local address 127.0.0.1:34220)
0928 14:44:49.046095 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:49.046108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:49.046195 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:49.046625 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:49.046627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:49.046648 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:49.046734 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.046739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.048289 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.048296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.049825 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.049832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.051564 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.051568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.051646 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.051671 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:44:49.051743 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:44:49.051809 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":152,"thread_start_us":73,"threads_started":1}
I20260928 14:44:49.052261 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.045761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.1.201.2:40113)
0928 14:44:49.046248 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:49.046259 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:49.046278 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:44:49.046346 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:49.046350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:49.046501 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:49.046643 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:49.046858 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.048136 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.049969 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.050811 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.050934 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.051125 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:49.051811 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 14:44:49.051914 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":144,"threads_started":1}
I20260928 14:44:49.053407 20671 tablet_service.cc:1467] Processing CreateTablet for tablet 89fdaff7797c4f16b317e8e0c6715b33 (DEFAULT_TABLE table=test-table [id=d8fed815466d46baa441338e9329a788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:49.054260 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.045248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.1.201.3:39981)
0928 14:44:49.046134 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:49.046146 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:49.046172 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:44:49.046250 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:49.046257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:49.046452 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:49.046934 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:49.047075 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.048248 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.049969 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.050811 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.050934 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.053967 (+ 3033us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:49.054026 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 14:44:49.054121 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":92,"threads_started":1}
I20260928 14:44:49.054490 20671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fdaff7797c4f16b317e8e0c6715b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:49.054826 20802 tablet_service.cc:1467] Processing CreateTablet for tablet 89fdaff7797c4f16b317e8e0c6715b33 (DEFAULT_TABLE table=test-table [id=d8fed815466d46baa441338e9329a788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:49.055752 20802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fdaff7797c4f16b317e8e0c6715b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:49.056984 20933 tablet_service.cc:1467] Processing CreateTablet for tablet 89fdaff7797c4f16b317e8e0c6715b33 (DEFAULT_TABLE table=test-table [id=d8fed815466d46baa441338e9329a788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:49.058005 20933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fdaff7797c4f16b317e8e0c6715b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:49.061328 21016 tablet_bootstrap.cc:492] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: Bootstrap starting.
I20260928 14:44:49.062739 21017 tablet_bootstrap.cc:492] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9: Bootstrap starting.
I20260928 14:44:49.063773 21016 tablet_bootstrap.cc:654] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:49.064445 21016 log.cc:824] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:49.064957 21017 tablet_bootstrap.cc:654] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:49.065112 21018 tablet_bootstrap.cc:492] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: Bootstrap starting.
I20260928 14:44:49.065721 21017 log.cc:824] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:49.066435 21016 tablet_bootstrap.cc:492] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: No bootstrap required, opened a new log
I20260928 14:44:49.066632 21016 ts_tablet_manager.cc:1397] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:44:49.067293 21018 tablet_bootstrap.cc:654] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:49.067529 21017 tablet_bootstrap.cc:492] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9: No bootstrap required, opened a new log
I20260928 14:44:49.067729 21017 ts_tablet_manager.cc:1397] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 14:44:49.068046 21018 log.cc:824] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:49.070058 21018 tablet_bootstrap.cc:492] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: No bootstrap required, opened a new log
I20260928 14:44:49.070242 21018 ts_tablet_manager.cc:1397] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:44:49.070876 21016 raft_consensus.cc:348] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 } } peers { permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 } } peers { permanent_uuid: "d4155064161a4552b246266e54f3b813" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40113 } }
I20260928 14:44:49.071147 21016 raft_consensus.cc:374] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:49.071209 21016 raft_consensus.cc:729] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4155064161a4552b246266e54f3b813, State: Initialized, Role: FOLLOWER
I20260928 14:44:49.071645 21016 consensus_queue.cc:260] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 } } peers { permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 } } peers { permanent_uuid: "d4155064161a4552b246266e54f3b813" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40113 } }
I20260928 14:44:49.071760 21017 raft_consensus.cc:348] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 } } peers { permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 } } peers { permanent_uuid: "d4155064161a4552b246266e54f3b813" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40113 } }
I20260928 14:44:49.072057 21017 raft_consensus.cc:374] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:49.072131 21017 raft_consensus.cc:729] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce815d4cddf4407a9b0b4bb495365b9, State: Initialized, Role: FOLLOWER
I20260928 14:44:49.072419 20868 heartbeater.cc:503] Master 127.1.201.62:41817 was elected leader, sending a full tablet report...
I20260928 14:44:49.072564 21016 ts_tablet_manager.cc:1428] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:44:49.072574 21017 consensus_queue.cc:260] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 } } peers { permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 } } peers { permanent_uuid: "d4155064161a4552b246266e54f3b813" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40113 } }
I20260928 14:44:49.073235 20737 heartbeater.cc:503] Master 127.1.201.62:41817 was elected leader, sending a full tablet report...
I20260928 14:44:49.073562 21017 ts_tablet_manager.cc:1428] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260928 14:44:49.074726 21018 raft_consensus.cc:348] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 } } peers { permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 } } peers { permanent_uuid: "d4155064161a4552b246266e54f3b813" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40113 } }
I20260928 14:44:49.074981 21018 raft_consensus.cc:374] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:49.075032 21018 raft_consensus.cc:729] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccc7e1fa24147178334a442c7dfb4fb, State: Initialized, Role: FOLLOWER
I20260928 14:44:49.075390 21018 consensus_queue.cc:260] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 } } peers { permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 } } peers { permanent_uuid: "d4155064161a4552b246266e54f3b813" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40113 } }
I20260928 14:44:49.076232 21018 ts_tablet_manager.cc:1428] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:44:49.076293 20999 heartbeater.cc:503] Master 127.1.201.62:41817 was elected leader, sending a full tablet report...
W20260928 14:44:49.078361 20869 tablet.cc:2369] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:49.097944 21022 raft_consensus.cc:482] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:49.098129 21022 raft_consensus.cc:504] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 } } peers { permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 } } peers { permanent_uuid: "d4155064161a4552b246266e54f3b813" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40113 } }
I20260928 14:44:49.099171 21022 leader_election.cc:288] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce815d4cddf4407a9b0b4bb495365b9 (127.1.201.1:44663), 3ccc7e1fa24147178334a442c7dfb4fb (127.1.201.3:39981)
I20260928 14:44:49.102881 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.099382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:39981 (local address 127.1.201.2:37573)
0928 14:44:49.099496 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:49.099513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:49.099599 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:49.099851 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:49.099853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:49.099871 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:49.100040 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.100048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.100913 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.100916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.101762 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.101771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.102472 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.102475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.102580 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.102601 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:44:49.102671 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:44:49.102739 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 14:44:49.102909 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.099491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:37573 (local address 127.1.201.3:39981)
0928 14:44:49.099614 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:49.099619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:49.099634 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:44:49.099686 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:49.099691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:49.099741 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:49.099822 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:49.100155 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.100804 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.101919 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.102363 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.102440 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.102484 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:49.102700 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 14:44:49.102777 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:44:49.103017 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.099477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:59291 (local address 127.1.201.1:44663)
0928 14:44:49.099622 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:49.099629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:49.099956 (+ 327us) server_negotiation.cc:407] Connection header received
0928 14:44:49.100062 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:49.100066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:49.100126 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:49.100203 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:49.100483 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.101023 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.102081 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.102479 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.102577 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.102634 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:49.102831 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 14:44:49.102909 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:44:49.103217 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.099382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:44663 (local address 127.1.201.2:59291)
0928 14:44:49.099860 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:49.099872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:49.099967 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:49.100229 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:49.100232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:49.100248 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:49.100375 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.100381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.101145 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.101148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.101967 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:49.101976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.102601 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.102604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:49.102708 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.102730 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:44:49.102801 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:44:49.102859 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":162,"thread_start_us":120,"threads_started":1}
I20260928 14:44:49.103601 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fdaff7797c4f16b317e8e0c6715b33" candidate_uuid: "d4155064161a4552b246266e54f3b813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" is_pre_election: true
I20260928 14:44:49.103803 20691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fdaff7797c4f16b317e8e0c6715b33" candidate_uuid: "d4155064161a4552b246266e54f3b813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce815d4cddf4407a9b0b4bb495365b9" is_pre_election: true
I20260928 14:44:49.103873 20953 raft_consensus.cc:2454] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4155064161a4552b246266e54f3b813 in term 0.
I20260928 14:44:49.104060 20691 raft_consensus.cc:2454] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4155064161a4552b246266e54f3b813 in term 0.
I20260928 14:44:49.104353 20757 leader_election.cc:302] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccc7e1fa24147178334a442c7dfb4fb, d4155064161a4552b246266e54f3b813; no voters:
I20260928 14:44:49.104591 21022 raft_consensus.cc:2781] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:44:49.104667 21022 raft_consensus.cc:482] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:44:49.104709 21022 raft_consensus.cc:3037] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:49.105659 21022 raft_consensus.cc:504] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 } } peers { permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 } } peers { permanent_uuid: "d4155064161a4552b246266e54f3b813" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40113 } }
I20260928 14:44:49.105989 21022 leader_election.cc:288] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [CANDIDATE]: Term 1 election: Requested vote from peers bce815d4cddf4407a9b0b4bb495365b9 (127.1.201.1:44663), 3ccc7e1fa24147178334a442c7dfb4fb (127.1.201.3:39981)
I20260928 14:44:49.106279 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fdaff7797c4f16b317e8e0c6715b33" candidate_uuid: "d4155064161a4552b246266e54f3b813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccc7e1fa24147178334a442c7dfb4fb"
I20260928 14:44:49.106293 20691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fdaff7797c4f16b317e8e0c6715b33" candidate_uuid: "d4155064161a4552b246266e54f3b813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce815d4cddf4407a9b0b4bb495365b9"
I20260928 14:44:49.106411 20691 raft_consensus.cc:3037] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:49.106383 20953 raft_consensus.cc:3037] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:49.107662 20953 raft_consensus.cc:2454] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4155064161a4552b246266e54f3b813 in term 1.
I20260928 14:44:49.107806 20691 raft_consensus.cc:2454] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4155064161a4552b246266e54f3b813 in term 1.
I20260928 14:44:49.108016 20757 leader_election.cc:302] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccc7e1fa24147178334a442c7dfb4fb, d4155064161a4552b246266e54f3b813; no voters:
I20260928 14:44:49.108194 21022 raft_consensus.cc:2781] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:49.108418 21022 raft_consensus.cc:686] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 1 LEADER]: Becoming Leader. State: Replica: d4155064161a4552b246266e54f3b813, State: Running, Role: LEADER
I20260928 14:44:49.108747 21022 consensus_queue.cc:237] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 } } peers { permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 } } peers { permanent_uuid: "d4155064161a4552b246266e54f3b813" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40113 } }
I20260928 14:44:49.111783 20549 catalog_manager.cc:5495] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4155064161a4552b246266e54f3b813 (127.1.201.2). New cstate: current_term: 1 leader_uuid: "d4155064161a4552b246266e54f3b813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4155064161a4552b246266e54f3b813" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20260928 14:44:49.130786 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.127176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.1.201.2:40113)
0928 14:44:49.127311 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:49.127316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:49.127328 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:44:49.127487 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:49.127489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:49.127538 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:49.127638 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:44:49.127894 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.128415 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.129301 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:49.129583 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:49.129684 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:49.130001 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:44:49.130393 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:44:49.130499 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:49.130577 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:44:49.130626 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:44:49.136445 20953 raft_consensus.cc:1261] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb [term 1 FOLLOWER]: Refusing update from remote peer d4155064161a4552b246266e54f3b813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:49.136445 20691 raft_consensus.cc:1261] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9 [term 1 FOLLOWER]: Refusing update from remote peer d4155064161a4552b246266e54f3b813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:49.137236 21026 consensus_queue.cc:1035] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce815d4cddf4407a9b0b4bb495365b9" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 44663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:49.137460 21022 consensus_queue.cc:1035] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccc7e1fa24147178334a442c7dfb4fb" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:49.144728 21031 mvcc.cc:205] Tried to move back new op lower bound from 7334324998695550976 to 7334324998592032768. Current Snapshot: MvccSnapshot[applied={T|T < 7334324998695550976}]
I20260928 14:44:49.153328 20548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:49.153497 20548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:49.155751 20548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f046e8f3e21494d8913744249e61185: Sending DeleteTablet for 3 replicas of tablet 89fdaff7797c4f16b317e8e0c6715b33
I20260928 14:44:49.156428 20671 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fdaff7797c4f16b317e8e0c6715b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:49 UTC) from {username='slave'} at 127.0.0.1:40838
I20260928 14:44:49.156469 20933 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fdaff7797c4f16b317e8e0c6715b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:49 UTC) from {username='slave'} at 127.0.0.1:36022
I20260928 14:44:49.156631 20802 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fdaff7797c4f16b317e8e0c6715b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:49 UTC) from {username='slave'} at 127.0.0.1:34220
I20260928 14:44:49.156864 21039 tablet_replica.cc:330] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: stopping tablet replica
I20260928 14:44:49.156864 21040 tablet_replica.cc:330] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9: stopping tablet replica
I20260928 14:44:49.157131 21040 raft_consensus.cc:2229] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:49.157131 21039 raft_consensus.cc:2229] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:49.157405 21039 raft_consensus.cc:2258] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:49.157405 21040 raft_consensus.cc:2258] T 89fdaff7797c4f16b317e8e0c6715b33 P bce815d4cddf4407a9b0b4bb495365b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:49.157913 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 20609
I20260928 14:44:49.158560 21041 tablet_replica.cc:330] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: stopping tablet replica
I20260928 14:44:49.158811 21041 raft_consensus.cc:2229] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:44:49.159152 21041 raft_consensus.cc:2258] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:49.159350 21039 ts_tablet_manager.cc:1905] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:49.160455 21041 ts_tablet_manager.cc:1905] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:49.161978 21039 ts_tablet_manager.cc:1918] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:49.162066 21039 log.cc:1196] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/89fdaff7797c4f16b317e8e0c6715b33
I20260928 14:44:49.162365 21039 ts_tablet_manager.cc:1939] T 89fdaff7797c4f16b317e8e0c6715b33 P 3ccc7e1fa24147178334a442c7dfb4fb: Deleting consensus metadata
I20260928 14:44:49.163125 21041 ts_tablet_manager.cc:1918] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:49.163210 21041 log.cc:1196] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/89fdaff7797c4f16b317e8e0c6715b33
I20260928 14:44:49.163493 21041 ts_tablet_manager.cc:1939] T 89fdaff7797c4f16b317e8e0c6715b33 P d4155064161a4552b246266e54f3b813: Deleting consensus metadata
I20260928 14:44:49.163614 20535 catalog_manager.cc:4841] TS 3ccc7e1fa24147178334a442c7dfb4fb (127.1.201.3:39981): tablet 89fdaff7797c4f16b317e8e0c6715b33 (table test-table [id=d8fed815466d46baa441338e9329a788]) successfully deleted
I20260928 14:44:49.165141 20535 catalog_manager.cc:4841] TS d4155064161a4552b246266e54f3b813 (127.1.201.2:40113): tablet 89fdaff7797c4f16b317e8e0c6715b33 (table test-table [id=d8fed815466d46baa441338e9329a788]) successfully deleted
W20260928 14:44:49.166919 20536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:44663 (error 108)
I20260928 14:44:49.167789 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:49.167522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:44663 (local address 127.0.0.1:40842)
0928 14:44:49.167661 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:49.167726 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 14:44:49.167929 20536 catalog_manager.cc:4584] TS bce815d4cddf4407a9b0b4bb495365b9 (127.1.201.1:44663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89fdaff7797c4f16b317e8e0c6715b33: Network error: Client connection negotiation failed: client connection to 127.1.201.1:44663: connect: Connection refused (error 111)
I20260928 14:44:49.168087 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 20740
I20260928 14:44:49.173563 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 20871
I20260928 14:44:49.179345 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 20518
2026-09-28T14:44:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (983 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:44:49 dist-test-slave-0n27 krb5kdc[21045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:44:49 dist-test-slave-0n27 krb5kdc[21045](info): set up 2 sockets
Sep 28 14:44:49 dist-test-slave-0n27 krb5kdc[21045](info): commencing operation
krb5kdc: starting...
W20260928 14:44:50.183370 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:44:50 dist-test-slave-0n27 krb5kdc[21045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:44:50.299124 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40637
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:40637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:50.407598 21058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:50.407845 21058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:50.407882 21058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:50.411861 21058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:50.411914 21058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:50.411945 21058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:50.411962 21058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:50.411975 21058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:50.417029 21058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:40637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:50.418023 21058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:50.418942 21058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:50.427271 21063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:50.427394 21058 server_base.cc:1034] running on GCE node
W20260928 14:44:50.427309 21064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:50.427309 21066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:50.428089 21058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:50.429184 21058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:50.430377 21058 hybrid_clock.cc:648] HybridClock initialized: now 1790606690430356 us; error 27 us; skew 500 ppm
Sep 28 14:44:50 dist-test-slave-0n27 krb5kdc[21045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:50.433028 21058 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:44:50.433913 21058 webserver.cc:460] Webserver started at http://127.1.201.62:35337/ using document root <none> and password file <none>
I20260928 14:44:50.434459 21058 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:50.434515 21058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:50.434676 21058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:50.436619 21058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "01107d9d7b214ab68808e33ca5aa18bc"
format_stamp: "Formatted at 2026-09-28 14:44:50 on dist-test-slave-0n27"
server_key: "4d17ff803982a0c13a98e677c5f3f63f"
server_key_iv: "ea9caad75cd2fa5ea23cbd8ec414f414"
server_key_version: "encryptionkey@0"
I20260928 14:44:50.437127 21058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "01107d9d7b214ab68808e33ca5aa18bc"
format_stamp: "Formatted at 2026-09-28 14:44:50 on dist-test-slave-0n27"
server_key: "4d17ff803982a0c13a98e677c5f3f63f"
server_key_iv: "ea9caad75cd2fa5ea23cbd8ec414f414"
server_key_version: "encryptionkey@0"
I20260928 14:44:50.440735 21058 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:44:50.443068 21073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:50.444330 21058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:44:50.444453 21058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "01107d9d7b214ab68808e33ca5aa18bc"
format_stamp: "Formatted at 2026-09-28 14:44:50 on dist-test-slave-0n27"
server_key: "4d17ff803982a0c13a98e677c5f3f63f"
server_key_iv: "ea9caad75cd2fa5ea23cbd8ec414f414"
server_key_version: "encryptionkey@0"
I20260928 14:44:50.444547 21058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:50.457159 21058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:50.459805 21058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:50.460028 21058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:50.466794 21058 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:40637
I20260928 14:44:50.466822 21125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:40637 every 8 connection(s)
I20260928 14:44:50.467686 21058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:50.470353 21126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:50.474635 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 21058
I20260928 14:44:50.474762 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:50.475035 1828 external_mini_cluster.cc:1428] Setting key 673dd5aa13a88aeb10b2cc5defd9dc15
I20260928 14:44:50.475538 21126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc: Bootstrap starting.
I20260928 14:44:50.477924 21126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:50.478664 21126 log.cc:824] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:50.480373 21126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc: No bootstrap required, opened a new log
Sep 28 14:44:50 dist-test-slave-0n27 krb5kdc[21045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:44:50.484092 21126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01107d9d7b214ab68808e33ca5aa18bc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40637 } }
I20260928 14:44:50.484351 21126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:50.484426 21126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01107d9d7b214ab68808e33ca5aa18bc, State: Initialized, Role: FOLLOWER
I20260928 14:44:50.484937 21126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01107d9d7b214ab68808e33ca5aa18bc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40637 } }
I20260928 14:44:50.485070 21126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:50.485155 21126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:50.485273 21126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:50.486159 21126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01107d9d7b214ab68808e33ca5aa18bc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40637 } }
I20260928 14:44:50.486481 21126 leader_election.cc:302] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01107d9d7b214ab68808e33ca5aa18bc; no voters:
I20260928 14:44:50.486754 21126 leader_election.cc:288] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:50.486836 21131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:50.487039 21131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [term 1 LEADER]: Becoming Leader. State: Replica: 01107d9d7b214ab68808e33ca5aa18bc, State: Running, Role: LEADER
I20260928 14:44:50.487385 21131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01107d9d7b214ab68808e33ca5aa18bc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40637 } }
I20260928 14:44:50.488018 21126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:50.489341 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:50.476348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.1.201.62:40637)
0928 14:44:50.476767 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:50.476793 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:50.476825 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:44:50.477396 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:50.477411 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:50.477720 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:50.478088 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:50.478937 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:50.479871 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:50.480731 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:50.481155 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:50.481306 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:50.483959 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:50.483964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:50.483976 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:50.484001 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:50.485846 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:50.486272 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:50.486280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:50.486287 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:50.486417 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:50.486700 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:50.486704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:50.486705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:50.487229 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:50.487423 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:50.488020 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 14:44:50.488330 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260928 14:44:50.490979 21133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01107d9d7b214ab68808e33ca5aa18bc. Latest consensus state: current_term: 1 leader_uuid: "01107d9d7b214ab68808e33ca5aa18bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01107d9d7b214ab68808e33ca5aa18bc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40637 } } }
I20260928 14:44:50.491096 21133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:50.491250 21132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01107d9d7b214ab68808e33ca5aa18bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01107d9d7b214ab68808e33ca5aa18bc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40637 } } }
I20260928 14:44:50.491345 21132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc [sys.catalog]: This master's current role is: LEADER
W20260928 14:44:50.493216 21145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:44:50.493271 21145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:44:50.493346 21147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:50.494273 21147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:50.499718 21147 catalog_manager.cc:1347] Generated new cluster ID: 4ce7e39acda0485f904a36ce34a55cc4
I20260928 14:44:50.499786 21147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:50.510581 21147 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:50.511652 21147 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:50.517360 21147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc: Generated new TSK 0
I20260928 14:44:50.518033 21147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.201.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
Entry for principal kudu/127.1.201.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab.
I20260928 14:44:50.548022 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40637
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:50.678421 21152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:50.678668 21152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:50.678709 21152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:50.682529 21152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:50.682571 21152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:50.682663 21152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:44:50.687449 21152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.1.keytab
--principal=kudu/127.1.201.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:50.688475 21152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:50.689634 21152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:50.698591 21158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:50.698625 21157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:50.698628 21160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:50.699178 21152 server_base.cc:1034] running on GCE node
I20260928 14:44:50.699663 21152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:50.700341 21152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:50.701519 21152 hybrid_clock.cc:648] HybridClock initialized: now 1790606690701501 us; error 28 us; skew 500 ppm
Sep 28 14:44:50 dist-test-slave-0n27 krb5kdc[21045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:50.704121 21152 init.cc:377] Logged in from keytab as kudu/127.1.201.1@KRBTEST.COM (short username kudu)
I20260928 14:44:50.704937 21152 webserver.cc:460] Webserver started at http://127.1.201.1:34325/ using document root <none> and password file <none>
I20260928 14:44:50.705461 21152 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:50.705549 21152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:50.705845 21152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:50.707701 21152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "7858f2d91cb34a05b1c50cd64bfb5950"
format_stamp: "Formatted at 2026-09-28 14:44:50 on dist-test-slave-0n27"
server_key: "5bd7d454a83fd0b703cb1fb4c4888e0e"
server_key_iv: "0efc9163532e350f85742a24da20a989"
server_key_version: "encryptionkey@0"
I20260928 14:44:50.708261 21152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "7858f2d91cb34a05b1c50cd64bfb5950"
format_stamp: "Formatted at 2026-09-28 14:44:50 on dist-test-slave-0n27"
server_key: "5bd7d454a83fd0b703cb1fb4c4888e0e"
server_key_iv: "0efc9163532e350f85742a24da20a989"
server_key_version: "encryptionkey@0"
I20260928 14:44:50.712049 21152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:44:50.714366 21167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:50.715512 21152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:50.715646 21152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "7858f2d91cb34a05b1c50cd64bfb5950"
format_stamp: "Formatted at 2026-09-28 14:44:50 on dist-test-slave-0n27"
server_key: "5bd7d454a83fd0b703cb1fb4c4888e0e"
server_key_iv: "0efc9163532e350f85742a24da20a989"
server_key_version: "encryptionkey@0"
I20260928 14:44:50.715786 21152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:50.747129 21152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:50.750345 21152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:50.750543 21152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:50.751123 21152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:50.751928 21152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:50.752005 21152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:50.752076 21152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:50.752125 21152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:50.760923 21152 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:42491
I20260928 14:44:50.760954 21280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:42491 every 8 connection(s)
I20260928 14:44:50.761803 21152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:44:50.764870 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 21152
I20260928 14:44:50.765039 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:44:50.765326 1828 external_mini_cluster.cc:1428] Setting key 71fdfe7e8215fa9d29e1359eeea2a424
Sep 28 14:44:50 dist-test-slave-0n27 krb5kdc[21045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.1@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.2@KRBTEST.COM; defaulting to no policy
I20260928 14:44:50.776250 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:50.763548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:53201 (local address 127.1.201.62:40637)
0928 14:44:50.763715 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:50.763719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:50.764571 (+ 852us) server_negotiation.cc:407] Connection header received
0928 14:44:50.766066 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:50.766069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:50.766123 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:50.766203 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:50.767328 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:50.767841 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:50.769605 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:50.769890 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:50.769973 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:50.772734 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:50.772743 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:50.772745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:50.772767 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:50.774227 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:50.774712 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:50.774718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:50.774722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:50.774801 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:50.775126 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:50.775130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:50.775132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:50.775309 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:50.775434 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:50.775942 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 14:44:50.776065 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":80}
I20260928 14:44:50.777100 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:50.763921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40637 (local address 127.1.201.1:53201)
0928 14:44:50.764411 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:50.764461 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:50.765864 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:50.766378 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:50.766384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:50.766742 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:50.767190 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:50.767202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:50.767966 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:50.767969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:50.769476 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:50.769491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:50.770037 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:50.770043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:50.770285 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:50.770884 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:50.770909 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:50.772562 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:50.774341 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:50.774347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:50.774360 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:50.774604 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:50.774897 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:50.774900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:50.774902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:50.775033 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:50.775434 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:50.775442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:50.775707 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 14:44:50.775897 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 14:44:50.776200 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.2@KRBTEST.COM" created.
I20260928 14:44:50.778448 21281 heartbeater.cc:347] Connected to a master server at 127.1.201.62:40637
I20260928 14:44:50.778689 21281 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:50.779161 21281 heartbeater.cc:511] Master 127.1.201.62:40637 requested a full tablet report, sending...
I20260928 14:44:50.780653 21090 ts_manager.cc:194] Registered new tserver with Master: 7858f2d91cb34a05b1c50cd64bfb5950 (127.1.201.1:42491)
I20260928 14:44:50.781876 21090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.1@KRBTEST.COM'} at 127.1.201.1:53201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
Entry for principal kudu/127.1.201.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab.
I20260928 14:44:50.792106 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40637
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:50.901296 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:50.901562 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:50.901638 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:50.905594 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:50.905665 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:50.905750 21286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:44:50.910733 21286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.2.keytab
--principal=kudu/127.1.201.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:50.911721 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:50.912751 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:50.922400 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:50.922413 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:50.922407 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:50.922452 21286 server_base.cc:1034] running on GCE node
I20260928 14:44:50.923050 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:50.923648 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:50.924835 21286 hybrid_clock.cc:648] HybridClock initialized: now 1790606690924821 us; error 28 us; skew 500 ppm
Sep 28 14:44:50 dist-test-slave-0n27 krb5kdc[21045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:50.927484 21286 init.cc:377] Logged in from keytab as kudu/127.1.201.2@KRBTEST.COM (short username kudu)
I20260928 14:44:50.928313 21286 webserver.cc:460] Webserver started at http://127.1.201.2:34925/ using document root <none> and password file <none>
I20260928 14:44:50.928861 21286 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:50.928951 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:50.929171 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:50.931170 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "7caa3b34b74040c6a9ddaf4a71772149"
format_stamp: "Formatted at 2026-09-28 14:44:50 on dist-test-slave-0n27"
server_key: "7e1c4b55357540525e716380a93a177c"
server_key_iv: "7990be7d8de36447ce62beaa7928ee4d"
server_key_version: "encryptionkey@0"
I20260928 14:44:50.931722 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "7caa3b34b74040c6a9ddaf4a71772149"
format_stamp: "Formatted at 2026-09-28 14:44:50 on dist-test-slave-0n27"
server_key: "7e1c4b55357540525e716380a93a177c"
server_key_iv: "7990be7d8de36447ce62beaa7928ee4d"
server_key_version: "encryptionkey@0"
I20260928 14:44:50.935158 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:44:50.937496 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:50.938736 21286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:50.938869 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "7caa3b34b74040c6a9ddaf4a71772149"
format_stamp: "Formatted at 2026-09-28 14:44:50 on dist-test-slave-0n27"
server_key: "7e1c4b55357540525e716380a93a177c"
server_key_iv: "7990be7d8de36447ce62beaa7928ee4d"
server_key_version: "encryptionkey@0"
I20260928 14:44:50.938977 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:50.951644 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:50.954771 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:50.954962 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:50.955518 21286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:50.956328 21286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:50.956396 21286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:50.956463 21286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:50.956508 21286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:50.964721 21286 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:34271
I20260928 14:44:50.964788 21414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:34271 every 8 connection(s)
I20260928 14:44:50.965556 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:44:50.968075 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 21286
I20260928 14:44:50.968222 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:44:50.968555 1828 external_mini_cluster.cc:1428] Setting key 5436617f1f5f6a78745b49aa83103d56
Sep 28 14:44:50 dist-test-slave-0n27 krb5kdc[21045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.2@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.3@KRBTEST.COM; defaulting to no policy
I20260928 14:44:50.980038 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:50.967242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:53237 (local address 127.1.201.62:40637)
0928 14:44:50.967398 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:50.967403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:50.968306 (+ 903us) server_negotiation.cc:407] Connection header received
0928 14:44:50.969891 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:50.969894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:50.969944 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:50.970031 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:50.971505 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:50.972250 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:50.973363 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:50.973798 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:50.973925 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:50.976468 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:50.976471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:50.976473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:50.976496 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:50.977952 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:50.978504 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:50.978508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:50.978510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:50.978593 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:50.978924 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:50.978928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:50.978930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:50.979104 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:50.979216 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:50.979752 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 14:44:50.979869 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.3@KRBTEST.COM" created.
I20260928 14:44:50.980865 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:50.967605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40637 (local address 127.1.201.2:53237)
0928 14:44:50.968132 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:50.968205 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:50.969638 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:50.970217 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:50.970226 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:50.970696 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:50.971360 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:50.971382 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:50.972376 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:50.972378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:50.973242 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:50.973251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:50.973936 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:50.973939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:50.974086 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:50.974735 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:50.974750 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:50.976299 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:50.978065 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:50.978070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:50.978084 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:50.978403 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:50.978693 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:50.978696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:50.978698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:50.978829 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:50.979209 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:50.979214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:50.979514 (+ 300us) client_negotiation.cc:769] Sending connection context
0928 14:44:50.979703 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 14:44:50.980001 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260928 14:44:50.982045 21415 heartbeater.cc:347] Connected to a master server at 127.1.201.62:40637
I20260928 14:44:50.982301 21415 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:50.982761 21415 heartbeater.cc:511] Master 127.1.201.62:40637 requested a full tablet report, sending...
I20260928 14:44:50.983807 21090 ts_manager.cc:194] Registered new tserver with Master: 7caa3b34b74040c6a9ddaf4a71772149 (127.1.201.2:34271)
I20260928 14:44:50.984377 21090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.2@KRBTEST.COM'} at 127.1.201.2:53237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
Entry for principal kudu/127.1.201.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab.
I20260928 14:44:50.994407 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40637
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:51.100839 21420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:51.101070 21420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:51.101109 21420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:51.104911 21420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:51.104962 21420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:51.105014 21420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:44:51.110054 21420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.3.keytab
--principal=kudu/127.1.201.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:51.110996 21420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:51.111920 21420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:44:51.121567 21420 server_base.cc:1034] running on GCE node
W20260928 14:44:51.121519 21425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:51.121524 21428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:51.121519 21426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:51.122206 21420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:51.122776 21420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:51.123971 21420 hybrid_clock.cc:648] HybridClock initialized: now 1790606691123962 us; error 29 us; skew 500 ppm
Sep 28 14:44:51 dist-test-slave-0n27 krb5kdc[21045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606691, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:51.126680 21420 init.cc:377] Logged in from keytab as kudu/127.1.201.3@KRBTEST.COM (short username kudu)
I20260928 14:44:51.127496 21420 webserver.cc:460] Webserver started at http://127.1.201.3:36671/ using document root <none> and password file <none>
I20260928 14:44:51.128039 21420 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:51.128127 21420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:51.128383 21420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:51.130456 21420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "0e87b4c8455a4e2d88c67e78a026ec1f"
format_stamp: "Formatted at 2026-09-28 14:44:51 on dist-test-slave-0n27"
server_key: "6780a52e0e95dd83a36a438d5ce646d9"
server_key_iv: "84bd1804a0260e80b059cbc0ea39d7b3"
server_key_version: "encryptionkey@0"
I20260928 14:44:51.131016 21420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "0e87b4c8455a4e2d88c67e78a026ec1f"
format_stamp: "Formatted at 2026-09-28 14:44:51 on dist-test-slave-0n27"
server_key: "6780a52e0e95dd83a36a438d5ce646d9"
server_key_iv: "84bd1804a0260e80b059cbc0ea39d7b3"
server_key_version: "encryptionkey@0"
I20260928 14:44:51.134622 21420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:44:51.136983 21435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:51.138273 21420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:44:51.138406 21420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "0e87b4c8455a4e2d88c67e78a026ec1f"
format_stamp: "Formatted at 2026-09-28 14:44:51 on dist-test-slave-0n27"
server_key: "6780a52e0e95dd83a36a438d5ce646d9"
server_key_iv: "84bd1804a0260e80b059cbc0ea39d7b3"
server_key_version: "encryptionkey@0"
I20260928 14:44:51.138516 21420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:51.161974 21420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:51.164891 21420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:51.165104 21420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:51.165697 21420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:51.166527 21420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:51.166599 21420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:51.166671 21420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:51.166714 21420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:51.175304 21420 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:39199
I20260928 14:44:51.175345 21548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:39199 every 8 connection(s)
I20260928 14:44:51.176167 21420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:44:51.179963 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 21420
I20260928 14:44:51.180120 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:44:51.180449 1828 external_mini_cluster.cc:1428] Setting key 4daa8f0424bff7a9894069a776cc6cf3
Sep 28 14:44:51 dist-test-slave-0n27 krb5kdc[21045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606691, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.3@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:44:51.191386 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.177906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:49335 (local address 127.1.201.62:40637)
0928 14:44:51.178054 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:51.178058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:51.178922 (+ 864us) server_negotiation.cc:407] Connection header received
0928 14:44:51.180221 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:51.180224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:51.180285 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:51.180384 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:51.181712 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.182315 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.183778 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.184123 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.184226 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.187561 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:51.187564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:51.187567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:51.187622 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:51.189292 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:51.189876 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:51.189879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:51.189881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:51.189973 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:51.190352 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:51.190355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:51.190356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:51.190538 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:51.190642 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:51.191109 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 14:44:51.191224 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:44:51.192157 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.178345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40637 (local address 127.1.201.3:49335)
0928 14:44:51.178776 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:51.178817 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:51.179962 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:51.180565 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:51.180573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:51.180955 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:51.181565 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.181581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.182482 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.182485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.183647 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.183657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.184393 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.184397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.184595 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.185230 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:44:51.185245 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:51.187393 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:51.189414 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:51.189420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:51.189435 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:51.189705 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:51.190084 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:44:51.190087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:44:51.190089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:44:51.190247 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:44:51.190640 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:51.190647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:51.190878 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 14:44:51.191053 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 14:44:51.191366 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260928 14:44:51.193272 21549 heartbeater.cc:347] Connected to a master server at 127.1.201.62:40637
I20260928 14:44:51.193506 21549 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:51.194046 21549 heartbeater.cc:511] Master 127.1.201.62:40637 requested a full tablet report, sending...
I20260928 14:44:51.195050 21090 ts_manager.cc:194] Registered new tserver with Master: 0e87b4c8455a4e2d88c67e78a026ec1f (127.1.201.3:39199)
I20260928 14:44:51.195555 21090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.201.3@KRBTEST.COM'} at 127.1.201.3:49335
I20260928 14:44:51.205870 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:44:51.215451 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.207740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.1.201.62:40637)
0928 14:44:51.207901 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:51.207905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:51.208046 (+ 141us) server_negotiation.cc:407] Connection header received
0928 14:44:51.208204 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:51.208208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:51.208282 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:51.208366 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:51.209098 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.209627 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.210524 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.210744 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.210845 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.212229 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:51.212232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:51.212235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:51.212260 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:51.213727 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:51.214200 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:51.214203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:51.214205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:51.214285 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:51.214559 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:51.214562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:51.214564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:51.214774 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:51.214912 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:51.215164 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 14:44:51.215270 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 14:44:51.218781 21090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:44:51.221899 21090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:44:51.235973 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.230563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.1.201.3:39199)
0928 14:44:51.231111 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:51.231130 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:51.231152 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:44:51.231227 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:51.231232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:51.231408 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:51.231628 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:51.231908 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.232976 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.234378 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.235209 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.235323 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.235461 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:51.235699 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 14:44:51.235807 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":166,"threads_started":1}
I20260928 14:44:51.235978 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.229610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:34271 (local address 127.0.0.1:33064)
0928 14:44:51.229912 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:51.229926 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:51.230020 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:51.231630 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:51.231633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:51.231676 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:51.231805 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.231809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.232881 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.232887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.234562 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.234569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.235460 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.235463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.235557 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.235602 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 14:44:51.235666 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:44:51.235809 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":319,"spinlock_wait_cycles":2688,"thread_start_us":85,"threads_started":1}
I20260928 14:44:51.235983 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.229521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42491 (local address 127.0.0.1:43542)
0928 14:44:51.229859 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:51.229904 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:51.230018 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:51.231267 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:51.231274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:51.231298 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:51.231421 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.231426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.232897 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.232900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.234406 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.234416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.235522 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.235526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.235679 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.235706 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:44:51.235782 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:44:51.235847 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":114,"thread_start_us":69,"threads_started":1}
I20260928 14:44:51.236371 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.230376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:39199 (local address 127.0.0.1:51520)
0928 14:44:51.230848 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:51.230861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:51.230946 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:51.231592 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:51.231596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:51.231617 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:51.231792 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.231799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.233105 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.233108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.234248 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.234257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.235339 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.235343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.235464 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.235506 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:44:51.235660 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 14:44:51.235809 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":67,"spinlock_wait_cycles":4992,"thread_start_us":97,"threads_started":1}
I20260928 14:44:51.236372 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.230344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.1.201.2:34271)
0928 14:44:51.231182 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:51.231194 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:51.231217 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:44:51.231286 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:51.231291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:51.231491 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:51.231923 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:51.232056 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.232754 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.234676 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.235330 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.235900 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.236052 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:51.236121 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 14:44:51.236222 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":116,"threads_started":1}
I20260928 14:44:51.238212 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.229921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.1.201.1:42491)
0928 14:44:51.230755 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:51.230788 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:51.230813 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:44:51.230890 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:51.230895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:51.231115 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:51.231275 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:51.231548 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.232746 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.234523 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.235234 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.235323 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.235461 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:51.237957 (+ 2496us) server_negotiation.cc:299] Negotiation successful
0928 14:44:51.238055 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":107,"threads_started":1}
I20260928 14:44:51.239164 21483 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff57bcc2a41420f9f405aabc219c66a (DEFAULT_TABLE table=test-table [id=7ebf418d11c54fd18430e3456c5024fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:51.239133 21349 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff57bcc2a41420f9f405aabc219c66a (DEFAULT_TABLE table=test-table [id=7ebf418d11c54fd18430e3456c5024fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:51.240120 21483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ff57bcc2a41420f9f405aabc219c66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:51.240150 21349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ff57bcc2a41420f9f405aabc219c66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:51.240784 21215 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff57bcc2a41420f9f405aabc219c66a (DEFAULT_TABLE table=test-table [id=7ebf418d11c54fd18430e3456c5024fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:51.241660 21215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ff57bcc2a41420f9f405aabc219c66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:51.246277 21566 tablet_bootstrap.cc:492] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: Bootstrap starting.
I20260928 14:44:51.247606 21567 tablet_bootstrap.cc:492] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: Bootstrap starting.
I20260928 14:44:51.248517 21568 tablet_bootstrap.cc:492] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950: Bootstrap starting.
I20260928 14:44:51.249204 21566 tablet_bootstrap.cc:654] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:51.249914 21567 tablet_bootstrap.cc:654] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:51.250365 21568 tablet_bootstrap.cc:654] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:51.250564 21566 log.cc:824] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:51.251091 21567 log.cc:824] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:51.252064 21568 log.cc:824] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:51.252542 21566 tablet_bootstrap.cc:492] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: No bootstrap required, opened a new log
I20260928 14:44:51.252730 21566 ts_tablet_manager.cc:1397] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 14:44:51.253002 21567 tablet_bootstrap.cc:492] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: No bootstrap required, opened a new log
I20260928 14:44:51.253176 21567 ts_tablet_manager.cc:1397] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260928 14:44:51.253703 21568 tablet_bootstrap.cc:492] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950: No bootstrap required, opened a new log
I20260928 14:44:51.253908 21568 ts_tablet_manager.cc:1397] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:44:51.256539 21566 raft_consensus.cc:348] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 } } peers { permanent_uuid: "7caa3b34b74040c6a9ddaf4a71772149" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34271 } } peers { permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 } }
I20260928 14:44:51.256742 21567 raft_consensus.cc:348] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 } } peers { permanent_uuid: "7caa3b34b74040c6a9ddaf4a71772149" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34271 } } peers { permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 } }
I20260928 14:44:51.256925 21566 raft_consensus.cc:374] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:51.256996 21567 raft_consensus.cc:374] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:51.256997 21566 raft_consensus.cc:729] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e87b4c8455a4e2d88c67e78a026ec1f, State: Initialized, Role: FOLLOWER
I20260928 14:44:51.257090 21567 raft_consensus.cc:729] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7caa3b34b74040c6a9ddaf4a71772149, State: Initialized, Role: FOLLOWER
I20260928 14:44:51.256935 21568 raft_consensus.cc:348] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 } } peers { permanent_uuid: "7caa3b34b74040c6a9ddaf4a71772149" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34271 } } peers { permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 } }
I20260928 14:44:51.257171 21568 raft_consensus.cc:374] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:51.257228 21568 raft_consensus.cc:729] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7858f2d91cb34a05b1c50cd64bfb5950, State: Initialized, Role: FOLLOWER
I20260928 14:44:51.257478 21567 consensus_queue.cc:260] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 } } peers { permanent_uuid: "7caa3b34b74040c6a9ddaf4a71772149" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34271 } } peers { permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 } }
I20260928 14:44:51.257507 21566 consensus_queue.cc:260] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 } } peers { permanent_uuid: "7caa3b34b74040c6a9ddaf4a71772149" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34271 } } peers { permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 } }
I20260928 14:44:51.257580 21568 consensus_queue.cc:260] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 } } peers { permanent_uuid: "7caa3b34b74040c6a9ddaf4a71772149" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34271 } } peers { permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 } }
I20260928 14:44:51.258273 21281 heartbeater.cc:503] Master 127.1.201.62:40637 was elected leader, sending a full tablet report...
I20260928 14:44:51.258474 21566 ts_tablet_manager.cc:1428] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:44:51.258474 21567 ts_tablet_manager.cc:1428] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260928 14:44:51.258814 21415 heartbeater.cc:503] Master 127.1.201.62:40637 was elected leader, sending a full tablet report...
I20260928 14:44:51.259637 21568 ts_tablet_manager.cc:1428] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:44:51.259703 21549 heartbeater.cc:503] Master 127.1.201.62:40637 was elected leader, sending a full tablet report...
W20260928 14:44:51.262893 21282 tablet.cc:2369] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:44:51.427109 21550 tablet.cc:2369] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:51.466262 21573 raft_consensus.cc:482] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:51.466492 21573 raft_consensus.cc:504] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 } } peers { permanent_uuid: "7caa3b34b74040c6a9ddaf4a71772149" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34271 } } peers { permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 } }
W20260928 14:44:51.466735 21416 tablet.cc:2369] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:51.467832 21573 leader_election.cc:288] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7858f2d91cb34a05b1c50cd64bfb5950 (127.1.201.1:42491), 0e87b4c8455a4e2d88c67e78a026ec1f (127.1.201.3:39199)
I20260928 14:44:51.472157 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.468044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:39199 (local address 127.1.201.2:37865)
0928 14:44:51.468171 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:51.468187 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:51.468288 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:51.468631 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:51.468634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:51.468652 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:51.468788 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.468794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.469835 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.469838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.470971 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.470982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.471739 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.471742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.471849 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.471873 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:44:51.471943 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:44:51.472029 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1}
I20260928 14:44:51.472164 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.468279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:37865 (local address 127.1.201.3:39199)
0928 14:44:51.468393 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:51.468398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:51.468412 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:44:51.468469 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:51.468475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:51.468531 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:51.468612 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:51.468918 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.469700 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.471089 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.471625 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.471725 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.471775 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:51.471978 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 14:44:51.472049 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:44:51.472187 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.468044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42491 (local address 127.1.201.2:55987)
0928 14:44:51.468666 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:51.468678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:51.468767 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:51.469004 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:51.469007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:51.469020 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:51.469141 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.469146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.470055 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.470059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.470857 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:51.470865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.471757 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.471760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:51.471866 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.471882 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 14:44:51.471967 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:44:51.472058 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":364,"thread_start_us":75,"threads_started":1}
I20260928 14:44:51.472599 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.468223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:55987 (local address 127.1.201.1:42491)
0928 14:44:51.468368 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:51.468373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:51.468749 (+ 376us) server_negotiation.cc:407] Connection header received
0928 14:44:51.468849 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:51.468855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:51.468910 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:51.468986 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:51.469252 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.469939 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.470981 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.471536 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.471625 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.471700 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:51.472438 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 14:44:51.472513 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:44:51.473222 21503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff57bcc2a41420f9f405aabc219c66a" candidate_uuid: "7caa3b34b74040c6a9ddaf4a71772149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" is_pre_election: true
I20260928 14:44:51.473209 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff57bcc2a41420f9f405aabc219c66a" candidate_uuid: "7caa3b34b74040c6a9ddaf4a71772149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" is_pre_election: true
I20260928 14:44:51.473506 21503 raft_consensus.cc:2454] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7caa3b34b74040c6a9ddaf4a71772149 in term 0.
I20260928 14:44:51.473506 21235 raft_consensus.cc:2454] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7caa3b34b74040c6a9ddaf4a71772149 in term 0.
I20260928 14:44:51.474052 21304 leader_election.cc:302] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e87b4c8455a4e2d88c67e78a026ec1f, 7858f2d91cb34a05b1c50cd64bfb5950, 7caa3b34b74040c6a9ddaf4a71772149; no voters:
I20260928 14:44:51.474334 21573 raft_consensus.cc:2781] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:44:51.474423 21573 raft_consensus.cc:482] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:44:51.474462 21573 raft_consensus.cc:3037] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:51.475694 21573 raft_consensus.cc:504] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 } } peers { permanent_uuid: "7caa3b34b74040c6a9ddaf4a71772149" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34271 } } peers { permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 } }
I20260928 14:44:51.476037 21573 leader_election.cc:288] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [CANDIDATE]: Term 1 election: Requested vote from peers 7858f2d91cb34a05b1c50cd64bfb5950 (127.1.201.1:42491), 0e87b4c8455a4e2d88c67e78a026ec1f (127.1.201.3:39199)
I20260928 14:44:51.476421 21503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff57bcc2a41420f9f405aabc219c66a" candidate_uuid: "7caa3b34b74040c6a9ddaf4a71772149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f"
I20260928 14:44:51.476539 21503 raft_consensus.cc:3037] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:51.477720 21503 raft_consensus.cc:2454] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7caa3b34b74040c6a9ddaf4a71772149 in term 1.
I20260928 14:44:51.478025 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff57bcc2a41420f9f405aabc219c66a" candidate_uuid: "7caa3b34b74040c6a9ddaf4a71772149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858f2d91cb34a05b1c50cd64bfb5950"
I20260928 14:44:51.478056 21302 leader_election.cc:302] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e87b4c8455a4e2d88c67e78a026ec1f, 7caa3b34b74040c6a9ddaf4a71772149; no voters:
I20260928 14:44:51.478149 21235 raft_consensus.cc:3037] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:51.479350 21235 raft_consensus.cc:2454] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7caa3b34b74040c6a9ddaf4a71772149 in term 1.
I20260928 14:44:51.479571 21573 raft_consensus.cc:2781] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:51.479796 21573 raft_consensus.cc:686] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 1 LEADER]: Becoming Leader. State: Replica: 7caa3b34b74040c6a9ddaf4a71772149, State: Running, Role: LEADER
I20260928 14:44:51.480108 21573 consensus_queue.cc:237] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 } } peers { permanent_uuid: "7caa3b34b74040c6a9ddaf4a71772149" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34271 } } peers { permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 } }
I20260928 14:44:51.483425 21089 catalog_manager.cc:5495] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7caa3b34b74040c6a9ddaf4a71772149 (127.1.201.2). New cstate: current_term: 1 leader_uuid: "7caa3b34b74040c6a9ddaf4a71772149" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7caa3b34b74040c6a9ddaf4a71772149" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 34271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 } health_report { overall_health: UNKNOWN } } }
I20260928 14:44:51.530758 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.527296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.1.201.2:34271)
0928 14:44:51.527429 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:51.527432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:51.527445 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:44:51.527541 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:51.527543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:51.527599 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:51.527687 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:44:51.527884 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.528395 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.529227 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.529440 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.529522 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.529766 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:44:51.530295 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:44:51.530384 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:51.530494 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 14:44:51.530540 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:44:51.536684 21235 raft_consensus.cc:1261] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950 [term 1 FOLLOWER]: Refusing update from remote peer 7caa3b34b74040c6a9ddaf4a71772149: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:51.536677 21503 raft_consensus.cc:1261] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f [term 1 FOLLOWER]: Refusing update from remote peer 7caa3b34b74040c6a9ddaf4a71772149: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:51.537405 21576 consensus_queue.cc:1035] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e87b4c8455a4e2d88c67e78a026ec1f" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:51.537600 21573 consensus_queue.cc:1035] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858f2d91cb34a05b1c50cd64bfb5950" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 42491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:51.545047 21579 mvcc.cc:205] Tried to move back new op lower bound from 7334325008526270464 to 7334325008305192960. Current Snapshot: MvccSnapshot[applied={T|T < 7334325008526270464}]
I20260928 14:44:51.547941 21581 mvcc.cc:205] Tried to move back new op lower bound from 7334325008526270464 to 7334325008305192960. Current Snapshot: MvccSnapshot[applied={T|T < 7334325008526270464}]
I20260928 14:44:51.554507 21089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:51.554646 21089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:51.557045 21089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01107d9d7b214ab68808e33ca5aa18bc: Sending DeleteTablet for 3 replicas of tablet 8ff57bcc2a41420f9f405aabc219c66a
I20260928 14:44:51.557806 21215 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ff57bcc2a41420f9f405aabc219c66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:51 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:43542
I20260928 14:44:51.557826 21349 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ff57bcc2a41420f9f405aabc219c66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:51 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:33064
I20260928 14:44:51.558085 21483 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ff57bcc2a41420f9f405aabc219c66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:51 UTC) from {username='kudu', principal='kudu/127.1.201.62@KRBTEST.COM'} at 127.0.0.1:51520
I20260928 14:44:51.558274 21589 tablet_replica.cc:330] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950: stopping tablet replica
I20260928 14:44:51.558583 21589 raft_consensus.cc:2229] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:51.558732 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 21152
I20260928 14:44:51.558943 21589 raft_consensus.cc:2258] T 8ff57bcc2a41420f9f405aabc219c66a P 7858f2d91cb34a05b1c50cd64bfb5950 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:51.560843 21591 tablet_replica.cc:330] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: stopping tablet replica
I20260928 14:44:51.561124 21591 raft_consensus.cc:2229] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:51.561414 21591 raft_consensus.cc:2258] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:51.570926 21590 tablet_replica.cc:330] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: stopping tablet replica
I20260928 14:44:51.571318 21590 raft_consensus.cc:2229] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:44:51.571740 21590 raft_consensus.cc:2258] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:44:51.573711 21076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:42491 (error 108)
I20260928 14:44:51.575649 21590 ts_tablet_manager.cc:1905] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:51.575659 21591 ts_tablet_manager.cc:1905] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:51.577754 21591 ts_tablet_manager.cc:1918] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:51.577960 21591 log.cc:1196] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/8ff57bcc2a41420f9f405aabc219c66a
W20260928 14:44:51.578114 21563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:44:51.574513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:42491 (local address 127.0.0.1:43546)
0928 14:44:51.574649 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:51.574665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:51.574754 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:51.577991 (+ 3237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:42491: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 14:44:51.578349 21076 catalog_manager.cc:4584] TS 7858f2d91cb34a05b1c50cd64bfb5950 (127.1.201.1:42491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ff57bcc2a41420f9f405aabc219c66a: Network error: Client connection negotiation failed: client connection to 127.1.201.1:42491: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:44:51.578615 21591 ts_tablet_manager.cc:1939] T 8ff57bcc2a41420f9f405aabc219c66a P 0e87b4c8455a4e2d88c67e78a026ec1f: Deleting consensus metadata
I20260928 14:44:51.578619 21590 ts_tablet_manager.cc:1918] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:51.578670 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 21286
I20260928 14:44:51.578699 21590 log.cc:1196] T 8ff57bcc2a41420f9f405aabc219c66a P 7caa3b34b74040c6a9ddaf4a71772149: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/8ff57bcc2a41420f9f405aabc219c66a
I20260928 14:44:51.581157 21074 catalog_manager.cc:4841] TS 0e87b4c8455a4e2d88c67e78a026ec1f (127.1.201.3:39199): tablet 8ff57bcc2a41420f9f405aabc219c66a (table test-table [id=7ebf418d11c54fd18430e3456c5024fb]) successfully deleted
I20260928 14:44:51.584960 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 21420
I20260928 14:44:51.585882 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.585084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:34271 (local address 127.0.0.1:33082)
0928 14:44:51.585639 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:51.585760 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.2:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460}
I20260928 14:44:51.590687 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 21058
2026-09-28T14:44:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2412 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T14:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:51Z Disabled control of system clock
I20260928 14:44:51.620812 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:44123
--webserver_interface=127.1.201.62
--webserver_port=0
--builtin_ntp_servers=127.1.201.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:44:51.727969 21596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:51.728246 21596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:51.728281 21596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:51.732074 21596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:51.732117 21596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:51.732132 21596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:51.732148 21596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:51.732162 21596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:51.736703 21596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:44123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:51.737664 21596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:51.738996 21596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:44:51.747330 21596 server_base.cc:1034] running on GCE node
W20260928 14:44:51.747304 21601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:51.747287 21602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:51.747287 21604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:51.748203 21596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:51.749274 21596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:51.750494 21596 hybrid_clock.cc:648] HybridClock initialized: now 1790606691750467 us; error 32 us; skew 500 ppm
I20260928 14:44:51.752353 21596 webserver.cc:460] Webserver started at http://127.1.201.62:40063/ using document root <none> and password file <none>
I20260928 14:44:51.752921 21596 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:51.753015 21596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:51.753252 21596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:51.755297 21596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "8c14191e1db04af18cff50cfe79e8976"
format_stamp: "Formatted at 2026-09-28 14:44:51 on dist-test-slave-0n27"
server_key: "552926e34015b4b8daacd464cbbe1025"
server_key_iv: "4add7c182c1e1d557fe6a80953dc2048"
server_key_version: "encryptionkey@0"
I20260928 14:44:51.755857 21596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "8c14191e1db04af18cff50cfe79e8976"
format_stamp: "Formatted at 2026-09-28 14:44:51 on dist-test-slave-0n27"
server_key: "552926e34015b4b8daacd464cbbe1025"
server_key_iv: "4add7c182c1e1d557fe6a80953dc2048"
server_key_version: "encryptionkey@0"
I20260928 14:44:51.759491 21596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:44:51.761737 21610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:51.762902 21596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:51.763033 21596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "8c14191e1db04af18cff50cfe79e8976"
format_stamp: "Formatted at 2026-09-28 14:44:51 on dist-test-slave-0n27"
server_key: "552926e34015b4b8daacd464cbbe1025"
server_key_iv: "4add7c182c1e1d557fe6a80953dc2048"
server_key_version: "encryptionkey@0"
I20260928 14:44:51.763135 21596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:51.778769 21596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:51.779388 21596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:51.779590 21596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:51.785948 21596 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:44123
I20260928 14:44:51.785961 21662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:44123 every 8 connection(s)
I20260928 14:44:51.786804 21596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:51.789842 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:51.795224 21663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976: Bootstrap starting.
I20260928 14:44:51.796051 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 21596
I20260928 14:44:51.796180 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:51.796526 1828 external_mini_cluster.cc:1428] Setting key 7f030cc96a3f9e92f086fe4ee1943a0f
I20260928 14:44:51.798194 21663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:51.798935 21663 log.cc:824] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:51.800712 21663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976: No bootstrap required, opened a new log
I20260928 14:44:51.804020 21663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c14191e1db04af18cff50cfe79e8976" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44123 } }
I20260928 14:44:51.804250 21663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:51.804338 21663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c14191e1db04af18cff50cfe79e8976, State: Initialized, Role: FOLLOWER
I20260928 14:44:51.804822 21663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c14191e1db04af18cff50cfe79e8976" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44123 } }
I20260928 14:44:51.804998 21663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:51.805052 21663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:51.805132 21663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:51.805569 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:51.797645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.1.201.62:44123)
0928 14:44:51.798338 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:51.798358 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:51.798402 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:44:51.799011 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:51.799027 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:51.799466 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:51.799809 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:51.800219 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.801200 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.802159 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:51.802630 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:51.802803 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:51.803643 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:51.803648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:51.803666 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:51.804036 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:51.804090 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:51.804110 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:51.804217 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:51.804362 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 14:44:51.804630 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":146,"threads_started":1}
I20260928 14:44:51.806190 21663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c14191e1db04af18cff50cfe79e8976" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44123 } }
I20260928 14:44:51.806543 21663 leader_election.cc:302] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c14191e1db04af18cff50cfe79e8976; no voters:
I20260928 14:44:51.806874 21663 leader_election.cc:288] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:51.806974 21668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:51.807176 21668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [term 1 LEADER]: Becoming Leader. State: Replica: 8c14191e1db04af18cff50cfe79e8976, State: Running, Role: LEADER
I20260928 14:44:51.807526 21668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c14191e1db04af18cff50cfe79e8976" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44123 } }
I20260928 14:44:51.808202 21663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:51.809372 21669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c14191e1db04af18cff50cfe79e8976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c14191e1db04af18cff50cfe79e8976" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44123 } } }
I20260928 14:44:51.809496 21669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:51.809756 21670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c14191e1db04af18cff50cfe79e8976. Latest consensus state: current_term: 1 leader_uuid: "8c14191e1db04af18cff50cfe79e8976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c14191e1db04af18cff50cfe79e8976" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 44123 } } }
I20260928 14:44:51.809903 21677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:51.809985 21670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:51.812960 21677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:51.818692 21677 catalog_manager.cc:1347] Generated new cluster ID: c85c00b888b94a24861836b3f9a41046
I20260928 14:44:51.818763 21677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:51.831955 21677 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:51.832772 21677 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:51.841635 21677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976: Generated new TSK 0
I20260928 14:44:51.842250 21677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:44:51.847266 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.1:0
--local_ip_for_outbound_sockets=127.1.201.1
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:44123
--builtin_ntp_servers=127.1.201.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:44:51.961696 21687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:51.961968 21687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:51.962038 21687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:51.966051 21687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:51.966117 21687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:51.966253 21687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.1
I20260928 14:44:51.971057 21687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.201.1
--webserver_port=0
--tserver_master_addrs=127.1.201.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.1
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:51.972059 21687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:51.973459 21687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:51.982414 21693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:51.982447 21692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:51.982445 21695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:51.982712 21687 server_base.cc:1034] running on GCE node
I20260928 14:44:51.983067 21687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:51.983651 21687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:51.984874 21687 hybrid_clock.cc:648] HybridClock initialized: now 1790606691984868 us; error 37 us; skew 500 ppm
I20260928 14:44:51.986760 21687 webserver.cc:460] Webserver started at http://127.1.201.1:44545/ using document root <none> and password file <none>
I20260928 14:44:51.987332 21687 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:51.987421 21687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:51.987646 21687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:51.989614 21687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/data/instance:
uuid: "436ff15aba1440029d0f1034f14a4734"
format_stamp: "Formatted at 2026-09-28 14:44:51 on dist-test-slave-0n27"
server_key: "894200831e03b7e08e11f30988dea34e"
server_key_iv: "191706eae2ddd75d5ac5959332aef580"
server_key_version: "encryptionkey@0"
I20260928 14:44:51.990224 21687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance:
uuid: "436ff15aba1440029d0f1034f14a4734"
format_stamp: "Formatted at 2026-09-28 14:44:51 on dist-test-slave-0n27"
server_key: "894200831e03b7e08e11f30988dea34e"
server_key_iv: "191706eae2ddd75d5ac5959332aef580"
server_key_version: "encryptionkey@0"
I20260928 14:44:51.993695 21687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:44:51.995916 21701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:51.997083 21687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:51.997222 21687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/wal
uuid: "436ff15aba1440029d0f1034f14a4734"
format_stamp: "Formatted at 2026-09-28 14:44:51 on dist-test-slave-0n27"
server_key: "894200831e03b7e08e11f30988dea34e"
server_key_iv: "191706eae2ddd75d5ac5959332aef580"
server_key_version: "encryptionkey@0"
I20260928 14:44:51.997328 21687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:52.007215 21687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:52.007817 21687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:52.007995 21687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:52.008566 21687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:52.009430 21687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:52.009500 21687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:52.009567 21687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:52.009598 21687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:52.018263 21687 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.1:38277
I20260928 14:44:52.018278 21814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.1:38277 every 8 connection(s)
I20260928 14:44:52.019168 21687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/data/info.pb
I20260928 14:44:52.023968 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 21687
I20260928 14:44:52.024070 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-0/wal/instance
I20260928 14:44:52.024377 1828 external_mini_cluster.cc:1428] Setting key a3682aa934299dcaa43bd923a2f48964
I20260928 14:44:52.026290 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.2:0
--local_ip_for_outbound_sockets=127.1.201.2
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:44123
--builtin_ntp_servers=127.1.201.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:44:52.028319 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.020884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.1:46111 (local address 127.1.201.62:44123)
0928 14:44:52.021026 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:52.021031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:52.022122 (+ 1091us) server_negotiation.cc:407] Connection header received
0928 14:44:52.023432 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:52.023434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:52.023478 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:52.023548 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:52.024441 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.025064 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.026355 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.026683 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.026793 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.027826 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:52.027829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:52.027832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:52.027844 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:52.027871 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:52.027880 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:52.027950 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:52.028107 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:44:52.028167 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:44:52.029101 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.021313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44123 (local address 127.1.201.1:46111)
0928 14:44:52.021983 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:52.022023 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:52.023253 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:52.023670 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:52.023676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:52.023755 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:52.024300 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.024316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.025192 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.025195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.026234 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.026244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.026824 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.026827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.026983 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.027575 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:44:52.027577 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:52.027614 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:44:52.027673 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:52.027964 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:52.027974 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:52.027986 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 14:44:52.028111 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 14:44:52.028379 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260928 14:44:52.030821 21815 heartbeater.cc:347] Connected to a master server at 127.1.201.62:44123
I20260928 14:44:52.031040 21815 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:52.031579 21815 heartbeater.cc:511] Master 127.1.201.62:44123 requested a full tablet report, sending...
I20260928 14:44:52.033032 21627 ts_manager.cc:194] Registered new tserver with Master: 436ff15aba1440029d0f1034f14a4734 (127.1.201.1:38277)
I20260928 14:44:52.034178 21627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.1:46111
W20260928 14:44:52.134120 21818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:52.134399 21818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:52.134469 21818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:52.138383 21818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:52.138465 21818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:52.138554 21818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.2
I20260928 14:44:52.143119 21818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.201.2
--webserver_port=0
--tserver_master_addrs=127.1.201.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.2
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:52.144109 21818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:52.145450 21818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:52.154769 21823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:52.154765 21826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:52.154794 21824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:52.155038 21818 server_base.cc:1034] running on GCE node
I20260928 14:44:52.155524 21818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:52.156113 21818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:52.157295 21818 hybrid_clock.cc:648] HybridClock initialized: now 1790606692157286 us; error 28 us; skew 500 ppm
I20260928 14:44:52.159189 21818 webserver.cc:460] Webserver started at http://127.1.201.2:33329/ using document root <none> and password file <none>
I20260928 14:44:52.159760 21818 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:52.159847 21818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:52.160064 21818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:52.162077 21818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/data/instance:
uuid: "a215800dda014ee7bf89c09c2599fff8"
format_stamp: "Formatted at 2026-09-28 14:44:52 on dist-test-slave-0n27"
server_key: "c42a8d9ff373aff703be38cdf439719c"
server_key_iv: "9a2e9bb2e7599ba8de7b2539ee07669a"
server_key_version: "encryptionkey@0"
I20260928 14:44:52.162597 21818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance:
uuid: "a215800dda014ee7bf89c09c2599fff8"
format_stamp: "Formatted at 2026-09-28 14:44:52 on dist-test-slave-0n27"
server_key: "c42a8d9ff373aff703be38cdf439719c"
server_key_iv: "9a2e9bb2e7599ba8de7b2539ee07669a"
server_key_version: "encryptionkey@0"
I20260928 14:44:52.166169 21818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:44:52.168478 21832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:52.169628 21818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:52.169756 21818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/wal
uuid: "a215800dda014ee7bf89c09c2599fff8"
format_stamp: "Formatted at 2026-09-28 14:44:52 on dist-test-slave-0n27"
server_key: "c42a8d9ff373aff703be38cdf439719c"
server_key_iv: "9a2e9bb2e7599ba8de7b2539ee07669a"
server_key_version: "encryptionkey@0"
I20260928 14:44:52.169907 21818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:52.196920 21818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:52.197598 21818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:52.197847 21818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:52.198446 21818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:52.199364 21818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:52.199431 21818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:52.199512 21818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:52.199558 21818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:52.208141 21818 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.2:41653
I20260928 14:44:52.208191 21945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.2:41653 every 8 connection(s)
I20260928 14:44:52.209079 21818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/data/info.pb
I20260928 14:44:52.212509 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 21818
I20260928 14:44:52.212638 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/wal/instance
I20260928 14:44:52.212935 1828 external_mini_cluster.cc:1428] Setting key ee00a7b5d95985dd299412e7de135bb6
I20260928 14:44:52.214996 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.201.3:0
--local_ip_for_outbound_sockets=127.1.201.3
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:44123
--builtin_ntp_servers=127.1.201.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:44:52.219002 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.211123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:43091 (local address 127.1.201.62:44123)
0928 14:44:52.211292 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:52.211297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:52.212402 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 14:44:52.214067 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:52.214069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:52.214114 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:52.214219 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:52.215222 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.215769 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.216894 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.217286 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.217405 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.218510 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:52.218514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:52.218517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:52.218536 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:52.218553 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:52.218561 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:52.218643 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:52.218799 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:44:52.218861 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:44:52.219832 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.211516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44123 (local address 127.1.201.2:43091)
0928 14:44:52.212244 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:52.212299 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:52.213878 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:52.214429 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:52.214436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:52.214521 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:52.215013 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.215026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.215903 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.215906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.216771 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.216780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.217434 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.217436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.217570 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.218215 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:44:52.218216 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:52.218256 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:44:52.218335 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:52.218645 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:52.218657 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:52.218665 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:44:52.218798 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 14:44:52.219067 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":137,"threads_started":1}
I20260928 14:44:52.221074 21946 heartbeater.cc:347] Connected to a master server at 127.1.201.62:44123
I20260928 14:44:52.221311 21946 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:52.221882 21946 heartbeater.cc:511] Master 127.1.201.62:44123 requested a full tablet report, sending...
I20260928 14:44:52.222976 21627 ts_manager.cc:194] Registered new tserver with Master: a215800dda014ee7bf89c09c2599fff8 (127.1.201.2:41653)
I20260928 14:44:52.223840 21627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.2:43091
W20260928 14:44:52.325363 21949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:52.325594 21949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:52.325628 21949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:52.329550 21949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:52.329599 21949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:52.329649 21949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.201.3
I20260928 14:44:52.334108 21949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.201.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.201.3
--webserver_port=0
--tserver_master_addrs=127.1.201.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.201.3
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:52.334998 21949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:52.336300 21949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:52.345839 21957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:52.345880 21954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:52.345849 21955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:52.346099 21949 server_base.cc:1034] running on GCE node
I20260928 14:44:52.346450 21949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:52.347064 21949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:52.348245 21949 hybrid_clock.cc:648] HybridClock initialized: now 1790606692348236 us; error 29 us; skew 500 ppm
I20260928 14:44:52.350173 21949 webserver.cc:460] Webserver started at http://127.1.201.3:36467/ using document root <none> and password file <none>
I20260928 14:44:52.350785 21949 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:52.350872 21949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:52.351094 21949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:52.353020 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/data/instance:
uuid: "26dc8a9b1ba24dcd9ff23060f5f91822"
format_stamp: "Formatted at 2026-09-28 14:44:52 on dist-test-slave-0n27"
server_key: "1414d910da4162e0cdc0d4b06ec56d4d"
server_key_iv: "ab39a73c7be0bbda7fa337691b8e1b3b"
server_key_version: "encryptionkey@0"
I20260928 14:44:52.353560 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance:
uuid: "26dc8a9b1ba24dcd9ff23060f5f91822"
format_stamp: "Formatted at 2026-09-28 14:44:52 on dist-test-slave-0n27"
server_key: "1414d910da4162e0cdc0d4b06ec56d4d"
server_key_iv: "ab39a73c7be0bbda7fa337691b8e1b3b"
server_key_version: "encryptionkey@0"
I20260928 14:44:52.356987 21949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:44:52.359293 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:52.360443 21949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:44:52.360574 21949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/wal
uuid: "26dc8a9b1ba24dcd9ff23060f5f91822"
format_stamp: "Formatted at 2026-09-28 14:44:52 on dist-test-slave-0n27"
server_key: "1414d910da4162e0cdc0d4b06ec56d4d"
server_key_iv: "ab39a73c7be0bbda7fa337691b8e1b3b"
server_key_version: "encryptionkey@0"
I20260928 14:44:52.360702 21949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:52.371551 21949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:52.372112 21949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:52.372282 21949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:52.372800 21949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:44:52.373723 21949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:44:52.373808 21949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:52.373878 21949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:44:52.373917 21949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:52.382377 21949 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.3:45203
I20260928 14:44:52.382403 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.3:45203 every 8 connection(s)
I20260928 14:44:52.383181 21949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/data/info.pb
I20260928 14:44:52.391491 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 21949
I20260928 14:44:52.391637 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/wal/instance
I20260928 14:44:52.391762 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.384815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.3:50035 (local address 127.1.201.62:44123)
0928 14:44:52.384940 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:52.384944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:52.386012 (+ 1068us) server_negotiation.cc:407] Connection header received
0928 14:44:52.387334 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:52.387337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:52.387381 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:52.387454 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:52.388260 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.388797 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.389885 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.390108 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.390196 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.391196 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:52.391202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:52.391204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:52.391219 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:52.391243 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:52.391278 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:52.391374 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:52.391515 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 14:44:52.391581 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:44:52.391978 1828 external_mini_cluster.cc:1428] Setting key 3e3ef33af06b48cae7eafe9a44ef4767
I20260928 14:44:52.392738 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.385199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:44123 (local address 127.1.201.3:50035)
0928 14:44:52.385813 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:52.385890 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:52.387100 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:52.387603 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:52.387609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:52.387688 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:44:52.388130 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.388140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.388899 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.388901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.389758 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.389766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.390259 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.390261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.390392 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.390921 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:44:52.390923 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:44:52.390959 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:44:52.391017 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:44:52.391379 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:44:52.391385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:44:52.391389 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:44:52.391478 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:44:52.391771 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":115,"threads_started":1}
I20260928 14:44:52.393796 22077 heartbeater.cc:347] Connected to a master server at 127.1.201.62:44123
I20260928 14:44:52.394016 22077 heartbeater.cc:464] Registering TS with master...
I20260928 14:44:52.394520 22077 heartbeater.cc:511] Master 127.1.201.62:44123 requested a full tablet report, sending...
I20260928 14:44:52.395393 21627 ts_manager.cc:194] Registered new tserver with Master: 26dc8a9b1ba24dcd9ff23060f5f91822 (127.1.201.3:45203)
I20260928 14:44:52.395967 21627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.201.3:50035
I20260928 14:44:52.405753 1828 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:44:52.411196 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.407521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.1.201.62:44123)
0928 14:44:52.407629 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:52.407633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:52.407842 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:44:52.407929 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:52.407931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:52.407966 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:52.408027 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:52.408235 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.408772 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.409633 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.409944 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.410055 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.410755 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:52.410759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:52.410762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:52.410773 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:52.410793 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:52.410809 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:52.410885 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:52.411004 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 14:44:52.411059 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:44:52.413766 21627 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:44:52.416669 21627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:44:52.430013 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.424376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.1.201.1:38277)
0928 14:44:52.424752 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:52.424766 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:52.424789 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:44:52.424869 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:52.424874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:52.425073 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:52.425237 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:52.425698 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.426913 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.428526 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.429318 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.429424 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.429625 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:52.429745 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 14:44:52.429859 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":156,"threads_started":1}
I20260928 14:44:52.430011 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.424205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:38277 (local address 127.0.0.1:41490)
0928 14:44:52.424504 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:52.424539 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:52.424688 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:52.425362 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:52.425370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:52.425407 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:52.425571 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.425581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.427011 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.427014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.428391 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.428401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.429433 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.429436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.429560 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.429597 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:44:52.429711 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 14:44:52.429859 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":22,"spinlock_wait_cycles":3584,"thread_start_us":68,"threads_started":1}
I20260928 14:44:52.430555 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.424833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:41653 (local address 127.0.0.1:41814)
0928 14:44:52.425299 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:52.425312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:52.425410 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:52.426300 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:52.426302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:52.426316 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:52.426389 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.426393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.427773 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.427777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.428903 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.428912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.430165 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.430168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.430250 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.430281 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:44:52.430348 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:44:52.430410 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":24,"spinlock_wait_cycles":4224,"thread_start_us":94,"threads_started":1}
I20260928 14:44:52.430647 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.425089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.1.201.2:41653)
0928 14:44:52.425641 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:52.425654 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:52.425676 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:44:52.425754 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:52.425758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:52.426020 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:52.426178 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:52.426521 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.427627 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.429023 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.429829 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.429930 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.430102 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:52.430395 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 14:44:52.430497 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":112,"threads_started":1}
I20260928 14:44:52.430863 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.425018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.1.201.3:45203)
0928 14:44:52.426340 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:52.426350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:52.426372 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:44:52.426459 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:52.426464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:52.426638 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:52.426763 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:52.427001 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.427837 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.429220 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.429992 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.430081 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.430250 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:52.430619 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 14:44:52.430721 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1234,"thread_start_us":135,"threads_started":1}
I20260928 14:44:52.431083 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.424833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:45203 (local address 127.0.0.1:56424)
0928 14:44:52.426024 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:52.426049 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:52.426126 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:52.426772 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:52.426775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:52.426794 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:52.426878 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.426884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.427957 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.427961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.429093 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.429103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.430303 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.430309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.430391 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.430409 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:44:52.430467 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:44:52.430517 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":201,"thread_start_us":70,"threads_started":1}
I20260928 14:44:52.433251 21749 tablet_service.cc:1467] Processing CreateTablet for tablet 90502af252aa4d2db74427aa73af916c (DEFAULT_TABLE table=test-table [id=836a77796edd4c61ae67b789a0d577f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:52.433544 22011 tablet_service.cc:1467] Processing CreateTablet for tablet 90502af252aa4d2db74427aa73af916c (DEFAULT_TABLE table=test-table [id=836a77796edd4c61ae67b789a0d577f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:52.433591 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 90502af252aa4d2db74427aa73af916c (DEFAULT_TABLE table=test-table [id=836a77796edd4c61ae67b789a0d577f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:44:52.434448 22011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90502af252aa4d2db74427aa73af916c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:52.434453 21749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90502af252aa4d2db74427aa73af916c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:52.434510 21880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90502af252aa4d2db74427aa73af916c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:52.441272 22095 tablet_bootstrap.cc:492] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: Bootstrap starting.
I20260928 14:44:52.441314 22094 tablet_bootstrap.cc:492] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: Bootstrap starting.
I20260928 14:44:52.441679 22096 tablet_bootstrap.cc:492] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734: Bootstrap starting.
I20260928 14:44:52.443981 22096 tablet_bootstrap.cc:654] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:52.443981 22094 tablet_bootstrap.cc:654] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:52.444871 22096 log.cc:824] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:52.444871 22094 log.cc:824] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:52.446882 22096 tablet_bootstrap.cc:492] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734: No bootstrap required, opened a new log
I20260928 14:44:52.447053 22095 tablet_bootstrap.cc:654] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:52.447067 22096 ts_tablet_manager.cc:1397] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:44:52.447124 22094 tablet_bootstrap.cc:492] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: No bootstrap required, opened a new log
I20260928 14:44:52.447335 22094 ts_tablet_manager.cc:1397] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:44:52.447768 22095 log.cc:824] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:52.449328 22095 tablet_bootstrap.cc:492] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: No bootstrap required, opened a new log
I20260928 14:44:52.449518 22095 ts_tablet_manager.cc:1397] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260928 14:44:52.451429 22096 raft_consensus.cc:348] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 } } peers { permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 } } peers { permanent_uuid: "a215800dda014ee7bf89c09c2599fff8" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 41653 } }
I20260928 14:44:52.451429 22094 raft_consensus.cc:348] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 } } peers { permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 } } peers { permanent_uuid: "a215800dda014ee7bf89c09c2599fff8" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 41653 } }
I20260928 14:44:52.451705 22094 raft_consensus.cc:374] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:52.451705 22096 raft_consensus.cc:374] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:52.451764 22096 raft_consensus.cc:729] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436ff15aba1440029d0f1034f14a4734, State: Initialized, Role: FOLLOWER
I20260928 14:44:52.451789 22094 raft_consensus.cc:729] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26dc8a9b1ba24dcd9ff23060f5f91822, State: Initialized, Role: FOLLOWER
I20260928 14:44:52.452244 22094 consensus_queue.cc:260] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 } } peers { permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 } } peers { permanent_uuid: "a215800dda014ee7bf89c09c2599fff8" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 41653 } }
I20260928 14:44:52.452244 22096 consensus_queue.cc:260] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 } } peers { permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 } } peers { permanent_uuid: "a215800dda014ee7bf89c09c2599fff8" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 41653 } }
I20260928 14:44:52.452896 21815 heartbeater.cc:503] Master 127.1.201.62:44123 was elected leader, sending a full tablet report...
I20260928 14:44:52.452867 22095 raft_consensus.cc:348] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 } } peers { permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 } } peers { permanent_uuid: "a215800dda014ee7bf89c09c2599fff8" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 41653 } }
I20260928 14:44:52.453118 22095 raft_consensus.cc:374] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:52.453184 22095 raft_consensus.cc:729] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a215800dda014ee7bf89c09c2599fff8, State: Initialized, Role: FOLLOWER
I20260928 14:44:52.453223 22096 ts_tablet_manager.cc:1428] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:44:52.453475 22077 heartbeater.cc:503] Master 127.1.201.62:44123 was elected leader, sending a full tablet report...
I20260928 14:44:52.453573 22095 consensus_queue.cc:260] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 } } peers { permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 } } peers { permanent_uuid: "a215800dda014ee7bf89c09c2599fff8" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 41653 } }
I20260928 14:44:52.454463 22094 ts_tablet_manager.cc:1428] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:44:52.454620 22095 ts_tablet_manager.cc:1428] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260928 14:44:52.455096 21946 heartbeater.cc:503] Master 127.1.201.62:44123 was elected leader, sending a full tablet report...
W20260928 14:44:52.460027 21947 tablet.cc:2369] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:44:52.520320 21816 tablet.cc:2369] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:44:52.634094 22078 tablet.cc:2369] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:44:52.752614 22102 raft_consensus.cc:482] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:44:52.752839 22102 raft_consensus.cc:504] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 } } peers { permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 } } peers { permanent_uuid: "a215800dda014ee7bf89c09c2599fff8" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 41653 } }
I20260928 14:44:52.754107 22102 leader_election.cc:288] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 436ff15aba1440029d0f1034f14a4734 (127.1.201.1:38277), 26dc8a9b1ba24dcd9ff23060f5f91822 (127.1.201.3:45203)
I20260928 14:44:52.757946 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.754512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:40573 (local address 127.1.201.1:38277)
0928 14:44:52.754662 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:52.754667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:52.754805 (+ 138us) server_negotiation.cc:407] Connection header received
0928 14:44:52.754938 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:52.754941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:52.755004 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:52.755098 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:52.755408 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.755924 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.757006 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.757389 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.757502 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.757561 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:52.757735 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:44:52.757826 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:44:52.757972 22103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.754382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:38277 (local address 127.1.201.2:40573)
0928 14:44:52.754690 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:52.754703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:52.754835 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:52.755110 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:52.755113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:52.755130 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:52.755275 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.755281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.756032 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.756037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.756888 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.756897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.757506 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.757509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.757596 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.757619 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:44:52.757720 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:44:52.757810 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":42,"spinlock_wait_cycles":2304,"thread_start_us":90,"threads_started":1}
I20260928 14:44:52.758167 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.754916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.201.2:42673 (local address 127.1.201.3:45203)
0928 14:44:52.755040 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:52.755044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:52.755128 (+ 84us) server_negotiation.cc:407] Connection header received
0928 14:44:52.755311 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:52.755313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:52.755370 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:52.755453 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:44:52.755737 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.756216 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.757239 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.757606 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.757729 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.757781 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:52.757990 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 14:44:52.758052 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:44:52.758201 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.754789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.3:45203 (local address 127.1.201.2:42673)
0928 14:44:52.755067 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:52.755079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:44:52.755178 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:44:52.755465 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:44:52.755468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:44:52.755484 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:44:52.755619 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.755625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.756324 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.756330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.757116 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:44:52.757124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.757739 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.757742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:44:52.757860 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.757882 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:44:52.757974 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:44:52.758040 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":4,"spinlock_wait_cycles":19072,"thread_start_us":62,"threads_started":1}
I20260928 14:44:52.758651 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90502af252aa4d2db74427aa73af916c" candidate_uuid: "a215800dda014ee7bf89c09c2599fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436ff15aba1440029d0f1034f14a4734" is_pre_election: true
I20260928 14:44:52.758731 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90502af252aa4d2db74427aa73af916c" candidate_uuid: "a215800dda014ee7bf89c09c2599fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" is_pre_election: true
I20260928 14:44:52.758975 21769 raft_consensus.cc:2454] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a215800dda014ee7bf89c09c2599fff8 in term 0.
I20260928 14:44:52.759016 22031 raft_consensus.cc:2454] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a215800dda014ee7bf89c09c2599fff8 in term 0.
I20260928 14:44:52.759475 21833 leader_election.cc:302] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26dc8a9b1ba24dcd9ff23060f5f91822, a215800dda014ee7bf89c09c2599fff8; no voters:
I20260928 14:44:52.759756 22102 raft_consensus.cc:2781] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:44:52.759840 22102 raft_consensus.cc:482] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:44:52.759882 22102 raft_consensus.cc:3037] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:52.760859 22102 raft_consensus.cc:504] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 } } peers { permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 } } peers { permanent_uuid: "a215800dda014ee7bf89c09c2599fff8" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 41653 } }
I20260928 14:44:52.761174 22102 leader_election.cc:288] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 436ff15aba1440029d0f1034f14a4734 (127.1.201.1:38277), 26dc8a9b1ba24dcd9ff23060f5f91822 (127.1.201.3:45203)
I20260928 14:44:52.761498 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90502af252aa4d2db74427aa73af916c" candidate_uuid: "a215800dda014ee7bf89c09c2599fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436ff15aba1440029d0f1034f14a4734"
I20260928 14:44:52.761507 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90502af252aa4d2db74427aa73af916c" candidate_uuid: "a215800dda014ee7bf89c09c2599fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822"
I20260928 14:44:52.761617 21769 raft_consensus.cc:3037] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:52.761617 22031 raft_consensus.cc:3037] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:52.762903 21769 raft_consensus.cc:2454] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a215800dda014ee7bf89c09c2599fff8 in term 1.
I20260928 14:44:52.762903 22031 raft_consensus.cc:2454] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a215800dda014ee7bf89c09c2599fff8 in term 1.
I20260928 14:44:52.763221 21833 leader_election.cc:302] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436ff15aba1440029d0f1034f14a4734, a215800dda014ee7bf89c09c2599fff8; no voters:
I20260928 14:44:52.763396 22102 raft_consensus.cc:2781] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:52.763661 22102 raft_consensus.cc:686] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 1 LEADER]: Becoming Leader. State: Replica: a215800dda014ee7bf89c09c2599fff8, State: Running, Role: LEADER
I20260928 14:44:52.764015 22102 consensus_queue.cc:237] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 } } peers { permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 } } peers { permanent_uuid: "a215800dda014ee7bf89c09c2599fff8" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 41653 } }
I20260928 14:44:52.767311 21627 catalog_manager.cc:5495] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a215800dda014ee7bf89c09c2599fff8 (127.1.201.2). New cstate: current_term: 1 leader_uuid: "a215800dda014ee7bf89c09c2599fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a215800dda014ee7bf89c09c2599fff8" member_type: VOTER last_known_addr { host: "127.1.201.2" port: 41653 } health_report { overall_health: HEALTHY } } }
I20260928 14:44:52.792825 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.788811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.1.201.2:41653)
0928 14:44:52.788934 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:52.788938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:52.788951 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:44:52.789111 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:52.789113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:52.789171 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:52.789253 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:44:52.789562 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.790172 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.791367 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:52.791575 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:52.791664 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:52.791975 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:44:52.792373 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:44:52.792472 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:52.792574 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:44:52.792646 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:44:52.798584 21769 raft_consensus.cc:1261] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734 [term 1 FOLLOWER]: Refusing update from remote peer a215800dda014ee7bf89c09c2599fff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:52.798594 22031 raft_consensus.cc:1261] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822 [term 1 FOLLOWER]: Refusing update from remote peer a215800dda014ee7bf89c09c2599fff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:44:52.799276 22102 consensus_queue.cc:1035] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436ff15aba1440029d0f1034f14a4734" member_type: VOTER last_known_addr { host: "127.1.201.1" port: 38277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:52.799551 22105 consensus_queue.cc:1035] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26dc8a9b1ba24dcd9ff23060f5f91822" member_type: VOTER last_known_addr { host: "127.1.201.3" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:44:52.817030 21625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:52.817205 21625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:44:52.819550 21625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c14191e1db04af18cff50cfe79e8976: Sending DeleteTablet for 3 replicas of tablet 90502af252aa4d2db74427aa73af916c
I20260928 14:44:52.820328 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 90502af252aa4d2db74427aa73af916c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:52 UTC) from {username='slave'} at 127.0.0.1:41814
I20260928 14:44:52.820328 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet 90502af252aa4d2db74427aa73af916c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:52 UTC) from {username='slave'} at 127.0.0.1:56424
I20260928 14:44:52.820371 21749 tablet_service.cc:1514] Processing DeleteTablet for tablet 90502af252aa4d2db74427aa73af916c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:44:52 UTC) from {username='slave'} at 127.0.0.1:41490
I20260928 14:44:52.820793 22120 tablet_replica.cc:330] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734: stopping tablet replica
I20260928 14:44:52.820916 22119 tablet_replica.cc:330] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: stopping tablet replica
I20260928 14:44:52.821111 22120 raft_consensus.cc:2229] T 90502af252aa4d2db74427aa73af916c P 436ff15aba1440029d0f1034f14a4734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:52.821219 22119 raft_consensus.cc:2229] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:44:52.821542 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 21687
I20260928 14:44:52.822322 22119 raft_consensus.cc:2258] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:52.822731 22118 tablet_replica.cc:330] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: stopping tablet replica
I20260928 14:44:52.822988 22118 raft_consensus.cc:2229] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:44:52.823366 22118 raft_consensus.cc:2258] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:44:52.824832 22118 ts_tablet_manager.cc:1905] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:52.826908 22119 ts_tablet_manager.cc:1905] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:44:52.828253 22118 ts_tablet_manager.cc:1918] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:52.828356 22118 log.cc:1196] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-1/wal/wals/90502af252aa4d2db74427aa73af916c
I20260928 14:44:52.828732 22118 ts_tablet_manager.cc:1939] T 90502af252aa4d2db74427aa73af916c P a215800dda014ee7bf89c09c2599fff8: Deleting consensus metadata
W20260928 14:44:52.828853 21611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.201.1:38277 (error 108)
I20260928 14:44:52.829052 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 21818
I20260928 14:44:52.830461 22119 ts_tablet_manager.cc:1918] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:44:52.830579 22119 log.cc:1196] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: Deleting WAL directory at /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606444849692-1828-0/minicluster-data/ts-2/wal/wals/90502af252aa4d2db74427aa73af916c
I20260928 14:44:52.830883 22119 ts_tablet_manager.cc:1939] T 90502af252aa4d2db74427aa73af916c P 26dc8a9b1ba24dcd9ff23060f5f91822: Deleting consensus metadata
I20260928 14:44:52.831310 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.831010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.1:38277 (local address 127.0.0.1:41506)
0928 14:44:52.831156 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:52.831231 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.1:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 14:44:52.831470 21611 catalog_manager.cc:4584] TS 436ff15aba1440029d0f1034f14a4734 (127.1.201.1:38277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90502af252aa4d2db74427aa73af916c: Network error: Client connection negotiation failed: client connection to 127.1.201.1:38277: connect: Connection refused (error 111)
I20260928 14:44:52.832049 21611 catalog_manager.cc:4841] TS 26dc8a9b1ba24dcd9ff23060f5f91822 (127.1.201.3:45203): tablet 90502af252aa4d2db74427aa73af916c (table test-table [id=836a77796edd4c61ae67b789a0d577f0]) successfully deleted
I20260928 14:44:52.835966 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 21949
I20260928 14:44:52.836742 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:52.836106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.2:41653 (local address 127.0.0.1:41836)
0928 14:44:52.836597 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 14:44:52.836655 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.2:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404}
I20260928 14:44:52.841331 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 21596
2026-09-28T14:44:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1249 ms)
[----------] 6 tests from EncryptionPolicyTest (10253 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:44:52 dist-test-slave-0n27 krb5kdc[22124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:44:52 dist-test-slave-0n27 krb5kdc[22124](info): set up 2 sockets
Sep 28 14:44:52 dist-test-slave-0n27 krb5kdc[22124](info): commencing operation
krb5kdc: starting...
W20260928 14:44:53.854772 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:44:53 dist-test-slave-0n27 krb5kdc[22124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:44:53.968954 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:54.075853 22137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:54.076118 22137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:54.076159 22137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:54.080152 22137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:54.080210 22137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:54.080229 22137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:54.080245 22137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:54.080261 22137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:54.085455 22137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:54.086506 22137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:54.087553 22137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:54.095979 22145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:54.096057 22143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:54.096067 22142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:54.096251 22137 server_base.cc:1034] running on GCE node
I20260928 14:44:54.096802 22137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:54.098009 22137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:54.099201 22137 hybrid_clock.cc:648] HybridClock initialized: now 1790606694099193 us; error 30 us; skew 500 ppm
Sep 28 14:44:54 dist-test-slave-0n27 krb5kdc[22124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:54.101924 22137 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:44:54.102746 22137 webserver.cc:460] Webserver started at http://127.0.0.1:33403/ using document root <none> and password file <none>
I20260928 14:44:54.103241 22137 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:54.103297 22137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:54.103461 22137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:54.105405 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "89f862a169504af4a07f5d5b0fd8b805"
format_stamp: "Formatted at 2026-09-28 14:44:54 on dist-test-slave-0n27"
server_key: "d776af0d3bd53a60b8899e308b06ad55"
server_key_iv: "4d759ebadb462a3dda9819854439eefb"
server_key_version: "encryptionkey@0"
I20260928 14:44:54.105969 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "89f862a169504af4a07f5d5b0fd8b805"
format_stamp: "Formatted at 2026-09-28 14:44:54 on dist-test-slave-0n27"
server_key: "d776af0d3bd53a60b8899e308b06ad55"
server_key_iv: "4d759ebadb462a3dda9819854439eefb"
server_key_version: "encryptionkey@0"
I20260928 14:44:54.109411 22137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:44:54.111749 22152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:54.112905 22137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:54.112999 22137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "89f862a169504af4a07f5d5b0fd8b805"
format_stamp: "Formatted at 2026-09-28 14:44:54 on dist-test-slave-0n27"
server_key: "d776af0d3bd53a60b8899e308b06ad55"
server_key_iv: "4d759ebadb462a3dda9819854439eefb"
server_key_version: "encryptionkey@0"
I20260928 14:44:54.113080 22137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:54.127794 22137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:54.130910 22137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:54.131053 22137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:54.138144 22137 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40113
I20260928 14:44:54.138168 22204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40113 every 8 connection(s)
I20260928 14:44:54.139060 22137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:54.141642 22205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:54.144451 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 22137
I20260928 14:44:54.144587 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:54.144871 1828 external_mini_cluster.cc:1428] Setting key fd5c852711ff104a92a3b41aa12c877f
I20260928 14:44:54.147460 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805: Bootstrap starting.
I20260928 14:44:54.149585 22205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:54.150365 22205 log.cc:824] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:54.152375 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805: No bootstrap required, opened a new log
Sep 28 14:44:54 dist-test-slave-0n27 krb5kdc[22124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:44:54.156152 22205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f862a169504af4a07f5d5b0fd8b805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } }
I20260928 14:44:54.156430 22205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:54.156493 22205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f862a169504af4a07f5d5b0fd8b805, State: Initialized, Role: FOLLOWER
I20260928 14:44:54.157024 22205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f862a169504af4a07f5d5b0fd8b805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } }
I20260928 14:44:54.157197 22205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:54.157279 22205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:54.157392 22205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:54.158509 22205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f862a169504af4a07f5d5b0fd8b805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } }
I20260928 14:44:54.158859 22205 leader_election.cc:302] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89f862a169504af4a07f5d5b0fd8b805; no voters:
I20260928 14:44:54.159186 22205 leader_election.cc:288] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:54.159262 22210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:54.159399 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:54.146286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35389 (local address 127.0.0.1:40113)
0928 14:44:54.146713 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:54.146727 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:54.146761 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:44:54.147370 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:54.147387 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:54.147752 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:54.148067 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:54.148877 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:54.149976 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:54.150953 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:54.151591 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:54.151765 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:54.154452 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:54.154457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:54.154468 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:54.154492 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:54.156151 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:54.156724 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:54.156732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:54.156741 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:54.156891 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:54.157358 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:54.157364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:54.157367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:54.157834 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:54.158007 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:54.158315 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 14:44:54.158530 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":130,"threads_started":1}
I20260928 14:44:54.159477 22210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [term 1 LEADER]: Becoming Leader. State: Replica: 89f862a169504af4a07f5d5b0fd8b805, State: Running, Role: LEADER
I20260928 14:44:54.160447 22205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:54.160602 22210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f862a169504af4a07f5d5b0fd8b805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } }
I20260928 14:44:54.162649 22212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89f862a169504af4a07f5d5b0fd8b805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f862a169504af4a07f5d5b0fd8b805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } } }
I20260928 14:44:54.162977 22212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:54.163069 22211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89f862a169504af4a07f5d5b0fd8b805. Latest consensus state: current_term: 1 leader_uuid: "89f862a169504af4a07f5d5b0fd8b805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f862a169504af4a07f5d5b0fd8b805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40113 } } }
I20260928 14:44:54.163156 22211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:54.163168 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 14:44:54.166891 22224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:44:54.166953 22224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:44:54.167478 22232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:54.168484 22232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:54.174423 22232 catalog_manager.cc:1347] Generated new cluster ID: c8eb8d9991ec4035a090fccb2423facc
I20260928 14:44:54.174561 22232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:54.176014 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:54.167093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57135 (local address 127.0.0.1:40113)
0928 14:44:54.167222 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:54.167227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:54.167562 (+ 335us) server_negotiation.cc:407] Connection header received
0928 14:44:54.167670 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:54.167674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:54.167724 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:54.167820 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:54.168609 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:54.169499 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:54.170466 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:54.170689 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:54.170797 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:54.172158 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:54.172161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:54.172163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:54.172194 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:54.174162 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:54.174782 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:54.174784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:54.174786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:54.174853 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:54.175187 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:54.175190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:54.175191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:54.175353 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:54.175499 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:54.175764 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 14:44:54.175863 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:44:54.206393 22232 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:54.207319 22232 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:54.213757 22232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89f862a169504af4a07f5d5b0fd8b805: Generated new TSK 0
I20260928 14:44:54.214485 22232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:44:54.223047 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 22137
W20260928 14:44:54.228938 22122 connection.cc:535] client connection to 127.0.0.1:40113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40113): Connection reset by peer (error 104)
2026-09-28T14:44:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:44:54 dist-test-slave-0n27 krb5kdc[22236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:44:54 dist-test-slave-0n27 krb5kdc[22236](info): set up 2 sockets
Sep 28 14:44:54 dist-test-slave-0n27 krb5kdc[22236](info): commencing operation
krb5kdc: starting...
W20260928 14:44:55.235092 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:44:55 dist-test-slave-0n27 krb5kdc[22236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:44:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:44:55.350502 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:55.461457 22249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:55.461705 22249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:55.461796 22249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:55.465664 22249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:55.465714 22249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:55.465734 22249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:55.465781 22249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:55.465839 22249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:55.470932 22249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:55.471972 22249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:55.473037 22249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:55.481253 22254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:55.481273 22255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:55.481308 22249 server_base.cc:1034] running on GCE node
W20260928 14:44:55.481271 22257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:55.482020 22249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:55.483039 22249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:55.484203 22249 hybrid_clock.cc:648] HybridClock initialized: now 1790606695484194 us; error 31 us; skew 500 ppm
Sep 28 14:44:55 dist-test-slave-0n27 krb5kdc[22236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:55.486892 22249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:44:55.487664 22249 webserver.cc:460] Webserver started at http://127.0.0.1:42017/ using document root <none> and password file <none>
I20260928 14:44:55.488147 22249 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:55.488210 22249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:55.488365 22249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:55.490329 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "006f2dbae6ff439f934c82b67cefcf08"
format_stamp: "Formatted at 2026-09-28 14:44:55 on dist-test-slave-0n27"
server_key: "6053283ef550581e245767206d49cea7"
server_key_iv: "aca75f186acc41052229974e4c82258a"
server_key_version: "encryptionkey@0"
I20260928 14:44:55.490818 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "006f2dbae6ff439f934c82b67cefcf08"
format_stamp: "Formatted at 2026-09-28 14:44:55 on dist-test-slave-0n27"
server_key: "6053283ef550581e245767206d49cea7"
server_key_iv: "aca75f186acc41052229974e4c82258a"
server_key_version: "encryptionkey@0"
I20260928 14:44:55.494293 22249 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:44:55.496595 22264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:55.497745 22249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:44:55.497908 22249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "006f2dbae6ff439f934c82b67cefcf08"
format_stamp: "Formatted at 2026-09-28 14:44:55 on dist-test-slave-0n27"
server_key: "6053283ef550581e245767206d49cea7"
server_key_iv: "aca75f186acc41052229974e4c82258a"
server_key_version: "encryptionkey@0"
I20260928 14:44:55.498000 22249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:55.515782 22249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:55.518497 22249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:55.518656 22249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:55.525477 22249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33135
I20260928 14:44:55.525517 22316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33135 every 8 connection(s)
I20260928 14:44:55.526501 22249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:55.529349 22317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:55.534693 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08: Bootstrap starting.
I20260928 14:44:55.535897 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 22249
I20260928 14:44:55.536033 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:55.536332 1828 external_mini_cluster.cc:1428] Setting key 4a790214df7a72340e7d4d0a4763e48d
I20260928 14:44:55.537441 22317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:55.538239 22317 log.cc:824] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:55.540200 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08: No bootstrap required, opened a new log
I20260928 14:44:55.544489 22317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006f2dbae6ff439f934c82b67cefcf08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } }
I20260928 14:44:55.544734 22317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:55.544816 22317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006f2dbae6ff439f934c82b67cefcf08, State: Initialized, Role: FOLLOWER
I20260928 14:44:55.545244 22317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006f2dbae6ff439f934c82b67cefcf08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } }
I20260928 14:44:55.545395 22317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:55.545482 22317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:55.545603 22317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:55.545742 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:55.537479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49821 (local address 127.0.0.1:33135)
0928 14:44:55.537933 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:55.537948 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:55.537981 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:44:55.538691 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:55.538708 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:55.539023 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:55.539282 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:55.539586 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:55.540596 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:55.541855 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:55.542315 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:55.542494 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:55.543399 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:55.543404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:55.543415 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:55.543859 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:55.543915 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:55.543938 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:55.544036 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:55.544374 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 14:44:55.544533 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":149,"threads_started":1}
I20260928 14:44:55.546729 22317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006f2dbae6ff439f934c82b67cefcf08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } }
I20260928 14:44:55.547063 22317 leader_election.cc:302] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 006f2dbae6ff439f934c82b67cefcf08; no voters:
I20260928 14:44:55.547364 22317 leader_election.cc:288] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:55.547451 22322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:55.547658 22322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [term 1 LEADER]: Becoming Leader. State: Replica: 006f2dbae6ff439f934c82b67cefcf08, State: Running, Role: LEADER
I20260928 14:44:55.548010 22322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006f2dbae6ff439f934c82b67cefcf08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } }
I20260928 14:44:55.548635 22317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:55.548965 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:44:55.549226 22322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 006f2dbae6ff439f934c82b67cefcf08. Latest consensus state: current_term: 1 leader_uuid: "006f2dbae6ff439f934c82b67cefcf08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006f2dbae6ff439f934c82b67cefcf08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } } }
I20260928 14:44:55.549342 22322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:55.549665 22323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "006f2dbae6ff439f934c82b67cefcf08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006f2dbae6ff439f934c82b67cefcf08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } } }
I20260928 14:44:55.549741 22323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:55.549759 22329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:55.552882 22329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:55.556430 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:55.551447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36053 (local address 127.0.0.1:33135)
0928 14:44:55.551543 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:55.551547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:55.551834 (+ 287us) server_negotiation.cc:407] Connection header received
0928 14:44:55.551987 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:55.551990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:55.552035 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:55.552114 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:55.552397 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:55.553155 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:55.554474 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:55.554855 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:55.554958 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:55.555927 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:55.555931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:55.555934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:55.555947 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:55.555975 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:55.555989 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:55.556076 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:55.556260 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 14:44:55.556297 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:44:55.558830 22329 catalog_manager.cc:1347] Generated new cluster ID: c74cecc03414456b90d2cd2c2542e02c
I20260928 14:44:55.558919 22329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:55.572329 22329 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:55.573256 22329 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:55.579347 22329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 006f2dbae6ff439f934c82b67cefcf08: Generated new TSK 0
I20260928 14:44:55.579970 22329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:44:55.588824 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 22249
W20260928 14:44:55.594229 22234 connection.cc:535] client connection to 127.0.0.1:33135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33135): Connection reset by peer (error 104)
2026-09-28T14:44:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:44:55 dist-test-slave-0n27 krb5kdc[22348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:44:55 dist-test-slave-0n27 krb5kdc[22348](info): set up 2 sockets
Sep 28 14:44:55 dist-test-slave-0n27 krb5kdc[22348](info): commencing operation
krb5kdc: starting...
W20260928 14:44:56.618921 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:44:56 dist-test-slave-0n27 krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:44:56.734678 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:56.842075 22361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:56.842355 22361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:56.842427 22361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:56.846402 22361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:56.846481 22361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:56.846518 22361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:56.846575 22361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:56.846611 22361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:56.851738 22361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44103
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:56.852808 22361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:56.853925 22361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:56.862294 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:56.862308 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:56.862371 22361 server_base.cc:1034] running on GCE node
W20260928 14:44:56.862294 22369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:56.863049 22361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:56.864195 22361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:56.865370 22361 hybrid_clock.cc:648] HybridClock initialized: now 1790606696865337 us; error 43 us; skew 500 ppm
Sep 28 14:44:56 dist-test-slave-0n27 krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:56.868235 22361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:44:56.869107 22361 webserver.cc:460] Webserver started at http://127.0.0.1:35437/ using document root <none> and password file <none>
I20260928 14:44:56.869738 22361 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:56.869853 22361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:56.870098 22361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:56.872123 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "b255fe594c8c4f8fa259b6a4dae61635"
format_stamp: "Formatted at 2026-09-28 14:44:56 on dist-test-slave-0n27"
server_key: "4525668b1e3e9ea835457dc8e0573db7"
server_key_iv: "6fbb8fc58454e74df2c8c4dcb329f1ec"
server_key_version: "encryptionkey@0"
I20260928 14:44:56.872713 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "b255fe594c8c4f8fa259b6a4dae61635"
format_stamp: "Formatted at 2026-09-28 14:44:56 on dist-test-slave-0n27"
server_key: "4525668b1e3e9ea835457dc8e0573db7"
server_key_iv: "6fbb8fc58454e74df2c8c4dcb329f1ec"
server_key_version: "encryptionkey@0"
I20260928 14:44:56.876377 22361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260928 14:44:56.878733 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:56.879992 22361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:56.880124 22361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "b255fe594c8c4f8fa259b6a4dae61635"
format_stamp: "Formatted at 2026-09-28 14:44:56 on dist-test-slave-0n27"
server_key: "4525668b1e3e9ea835457dc8e0573db7"
server_key_iv: "6fbb8fc58454e74df2c8c4dcb329f1ec"
server_key_version: "encryptionkey@0"
I20260928 14:44:56.880260 22361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:56.884579 22361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:56.887734 22361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:56.887930 22361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:56.894626 22361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44103
I20260928 14:44:56.894680 22428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44103 every 8 connection(s)
I20260928 14:44:56.895522 22361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:56.898254 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:56.900182 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 22361
I20260928 14:44:56.900341 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:56.900624 1828 external_mini_cluster.cc:1428] Setting key 6f0f4ca13414b4821f6f57e2ca7d179d
I20260928 14:44:56.904381 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635: Bootstrap starting.
I20260928 14:44:56.906018 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:56.902006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51845 (local address 127.0.0.1:44103)
0928 14:44:56.902416 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:56.902434 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:56.902475 (+ 41us) server_negotiation.cc:407] Connection header received
0928 14:44:56.903335 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:56.903358 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:56.903695 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:56.904031 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:56.904699 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:56.904704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:56.904718 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:56.904730 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:56.904789 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:56.904813 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:56.904939 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:56.905002 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 14:44:56.905135 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260928 14:44:56.906996 22429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:56.907727 22429 log.cc:824] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:56.909027 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:44:56.909765 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635: No bootstrap required, opened a new log
I20260928 14:44:56.912628 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:56.911102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33909 (local address 127.0.0.1:44103)
0928 14:44:56.911224 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:56.911228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:56.911400 (+ 172us) server_negotiation.cc:407] Connection header received
0928 14:44:56.911527 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:56.911529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:56.911554 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:56.911617 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:56.912271 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:56.912274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:56.912275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:56.912285 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:56.912308 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:56.912317 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:56.912383 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:56.912486 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 14:44:56.912513 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:44:56.914090 22429 raft_consensus.cc:348] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255fe594c8c4f8fa259b6a4dae61635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44103 } }
I20260928 14:44:56.914326 22429 raft_consensus.cc:374] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:56.914399 22429 raft_consensus.cc:729] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b255fe594c8c4f8fa259b6a4dae61635, State: Initialized, Role: FOLLOWER
I20260928 14:44:56.914780 22429 consensus_queue.cc:260] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255fe594c8c4f8fa259b6a4dae61635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44103 } }
I20260928 14:44:56.914925 22429 raft_consensus.cc:388] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:56.915011 22429 raft_consensus.cc:482] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:56.915104 22429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:56.915927 22429 raft_consensus.cc:504] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255fe594c8c4f8fa259b6a4dae61635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44103 } }
I20260928 14:44:56.916297 22429 leader_election.cc:302] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b255fe594c8c4f8fa259b6a4dae61635; no voters:
I20260928 14:44:56.916604 22429 leader_election.cc:288] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:56.916700 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:56.916913 22439 raft_consensus.cc:686] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [term 1 LEADER]: Becoming Leader. State: Replica: b255fe594c8c4f8fa259b6a4dae61635, State: Running, Role: LEADER
I20260928 14:44:56.917222 22439 consensus_queue.cc:237] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255fe594c8c4f8fa259b6a4dae61635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44103 } }
I20260928 14:44:56.917765 22429 sys_catalog.cc:564] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:56.918888 22441 sys_catalog.cc:455] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b255fe594c8c4f8fa259b6a4dae61635. Latest consensus state: current_term: 1 leader_uuid: "b255fe594c8c4f8fa259b6a4dae61635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255fe594c8c4f8fa259b6a4dae61635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44103 } } }
I20260928 14:44:56.918910 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b255fe594c8c4f8fa259b6a4dae61635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255fe594c8c4f8fa259b6a4dae61635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44103 } } }
I20260928 14:44:56.919018 22441 sys_catalog.cc:458] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:56.919029 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:56.920001 22448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:56.922818 22448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:56.927376 22448 catalog_manager.cc:1347] Generated new cluster ID: e6cadc14b33245e7bd7c850035313bc5
I20260928 14:44:56.927430 22448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:56.953892 22448 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:56.954239 22448 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:56.961714 22448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b255fe594c8c4f8fa259b6a4dae61635: Generated new TSK 0
I20260928 14:44:56.962471 22448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:44:56.974844 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 22361
2026-09-28T14:44:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:44:57 dist-test-slave-0n27 krb5kdc[22460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:44:57 dist-test-slave-0n27 krb5kdc[22460](info): set up 2 sockets
Sep 28 14:44:57 dist-test-slave-0n27 krb5kdc[22460](info): commencing operation
krb5kdc: starting...
W20260928 14:44:57.978818 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:44:58 dist-test-slave-0n27 krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:44:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:44:58.094820 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:58.201771 22473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:58.202142 22473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:58.202224 22473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:58.206192 22473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:58.206282 22473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:58.206321 22473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:58.206377 22473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:58.206416 22473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:58.212563 22473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:58.213866 22473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:58.215065 22473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:58.223660 22479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:58.223604 22481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:58.223599 22478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:58.223839 22473 server_base.cc:1034] running on GCE node
I20260928 14:44:58.224515 22473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:58.225622 22473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:58.226816 22473 hybrid_clock.cc:648] HybridClock initialized: now 1790606698226800 us; error 37 us; skew 500 ppm
Sep 28 14:44:58 dist-test-slave-0n27 krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:58.235175 22473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:44:58.236083 22473 webserver.cc:460] Webserver started at http://127.0.0.1:34881/ using document root <none> and password file <none>
I20260928 14:44:58.236635 22473 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:58.236696 22473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:58.236941 22473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:58.239048 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "f9d1965dff7c496e9ef2faa819814481"
format_stamp: "Formatted at 2026-09-28 14:44:58 on dist-test-slave-0n27"
server_key: "4f896893c31fd86011958a18b3275fac"
server_key_iv: "143ae6c324e66bc41ae9af1068cfca8e"
server_key_version: "encryptionkey@0"
I20260928 14:44:58.239611 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "f9d1965dff7c496e9ef2faa819814481"
format_stamp: "Formatted at 2026-09-28 14:44:58 on dist-test-slave-0n27"
server_key: "4f896893c31fd86011958a18b3275fac"
server_key_iv: "143ae6c324e66bc41ae9af1068cfca8e"
server_key_version: "encryptionkey@0"
I20260928 14:44:58.243170 22473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:44:58.245385 22488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:58.246579 22473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:58.246711 22473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "f9d1965dff7c496e9ef2faa819814481"
format_stamp: "Formatted at 2026-09-28 14:44:58 on dist-test-slave-0n27"
server_key: "4f896893c31fd86011958a18b3275fac"
server_key_iv: "143ae6c324e66bc41ae9af1068cfca8e"
server_key_version: "encryptionkey@0"
I20260928 14:44:58.246816 22473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:58.259387 22473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:58.262478 22473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:58.262662 22473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:58.269474 22473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46641
I20260928 14:44:58.269500 22540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46641 every 8 connection(s)
I20260928 14:44:58.270392 22473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:58.273020 22541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:58.278087 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481: Bootstrap starting.
I20260928 14:44:58.280391 22541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:58.280544 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 22473
I20260928 14:44:58.280692 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:58.280974 1828 external_mini_cluster.cc:1428] Setting key 65a342b9e935f24a3bbfa032990d7586
I20260928 14:44:58.281250 22541 log.cc:824] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:58.283170 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481: No bootstrap required, opened a new log
I20260928 14:44:58.287295 22541 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d1965dff7c496e9ef2faa819814481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46641 } }
I20260928 14:44:58.287556 22541 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:58.287621 22541 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d1965dff7c496e9ef2faa819814481, State: Initialized, Role: FOLLOWER
I20260928 14:44:58.288142 22541 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d1965dff7c496e9ef2faa819814481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46641 } }
I20260928 14:44:58.288328 22541 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:58.288399 22541 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:58.288523 22541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 14:44:58 dist-test-slave-0n27 krb5kdc[22460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:44:58.289597 22541 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d1965dff7c496e9ef2faa819814481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46641 } }
I20260928 14:44:58.289948 22541 leader_election.cc:302] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9d1965dff7c496e9ef2faa819814481; no voters:
I20260928 14:44:58.290272 22541 leader_election.cc:288] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:58.290331 22546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:58.290534 22546 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [term 1 LEADER]: Becoming Leader. State: Replica: f9d1965dff7c496e9ef2faa819814481, State: Running, Role: LEADER
I20260928 14:44:58.290866 22546 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d1965dff7c496e9ef2faa819814481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46641 } }
I20260928 14:44:58.291507 22541 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:58.292466 22547 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9d1965dff7c496e9ef2faa819814481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d1965dff7c496e9ef2faa819814481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46641 } } }
I20260928 14:44:58.292590 22547 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:58.292933 22548 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9d1965dff7c496e9ef2faa819814481. Latest consensus state: current_term: 1 leader_uuid: "f9d1965dff7c496e9ef2faa819814481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d1965dff7c496e9ef2faa819814481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46641 } } }
I20260928 14:44:58.293089 22548 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481 [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:58.293649 22555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:58.295799 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:58.282107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49601 (local address 127.0.0.1:46641)
0928 14:44:58.282462 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:58.282477 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:58.282510 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:44:58.283095 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:58.283129 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:58.283490 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:58.283769 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:58.284630 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:58.285583 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:58.286885 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:58.287345 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:58.287463 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:58.290320 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:58.290344 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:58.290362 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:58.290395 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:58.292358 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:58.292862 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:58.292867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:58.292873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:58.293001 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:58.293313 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:58.293318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:58.293321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:58.293823 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:58.294506 (+ 683us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:58.294643 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 14:44:58.294937 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260928 14:44:58.296679 22555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:58.298827 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:44:58.302570 22555 catalog_manager.cc:1347] Generated new cluster ID: 5a0fd44e23604aec99d6a0513675e4a8
I20260928 14:44:58.302627 22555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:58.309063 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:58.300787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53875 (local address 127.0.0.1:46641)
0928 14:44:58.300884 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:58.300889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:58.301339 (+ 450us) server_negotiation.cc:407] Connection header received
0928 14:44:58.301438 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:58.301441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:58.301486 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:58.301566 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:58.302344 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:58.303260 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:58.304039 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:58.304276 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:58.304383 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:58.305823 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:58.305827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:58.305830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:44:58.305853 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:58.307227 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:58.307637 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:58.307641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:58.307643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:58.307722 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:44:58.307990 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:44:58.307994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:44:58.307996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:44:58.308170 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:58.308306 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:58.308832 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 14:44:58.308958 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:44:58.321214 22555 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:58.322387 22555 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:58.333590 22555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9d1965dff7c496e9ef2faa819814481: Generated new TSK 0
I20260928 14:44:58.334282 22555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:44:58.342880 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 22473
2026-09-28T14:44:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:44:58 dist-test-slave-0n27 krb5kdc[22572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:44:58 dist-test-slave-0n27 krb5kdc[22572](info): set up 2 sockets
Sep 28 14:44:58 dist-test-slave-0n27 krb5kdc[22572](info): commencing operation
krb5kdc: starting...
W20260928 14:44:59.351101 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:44:59 dist-test-slave-0n27 krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:44:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:44:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:44:59.468469 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:44:59.580531 22585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:44:59.580789 22585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:44:59.580830 22585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:44:59.584774 22585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:44:59.584828 22585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:44:59.584847 22585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:44:59.584863 22585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:44:59.584878 22585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:44:59.589977 22585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:44:59.590925 22585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:44:59.591861 22585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:44:59.600096 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:59.600104 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:44:59.600171 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:44:59.600543 22585 server_base.cc:1034] running on GCE node
I20260928 14:44:59.601114 22585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:44:59.602262 22585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:44:59.603451 22585 hybrid_clock.cc:648] HybridClock initialized: now 1790606699603433 us; error 29 us; skew 500 ppm
Sep 28 14:44:59 dist-test-slave-0n27 krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:44:59.606035 22585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:44:59.606971 22585 webserver.cc:460] Webserver started at http://127.0.0.1:38805/ using document root <none> and password file <none>
I20260928 14:44:59.607468 22585 fs_manager.cc:359] Metadata directory not provided
I20260928 14:44:59.607525 22585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:44:59.607681 22585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:44:59.609607 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "9dc5897fd9494a6995537dfb5aabcb5b"
format_stamp: "Formatted at 2026-09-28 14:44:59 on dist-test-slave-0n27"
server_key: "f2875bf2c1f829ec21158c968c784c33"
server_key_iv: "2914c50ef8686505abb8a7f05d128332"
server_key_version: "encryptionkey@0"
I20260928 14:44:59.610181 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "9dc5897fd9494a6995537dfb5aabcb5b"
format_stamp: "Formatted at 2026-09-28 14:44:59 on dist-test-slave-0n27"
server_key: "f2875bf2c1f829ec21158c968c784c33"
server_key_iv: "2914c50ef8686505abb8a7f05d128332"
server_key_version: "encryptionkey@0"
I20260928 14:44:59.613847 22585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:44:59.616110 22600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:44:59.617246 22585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:44:59.617370 22585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "9dc5897fd9494a6995537dfb5aabcb5b"
format_stamp: "Formatted at 2026-09-28 14:44:59 on dist-test-slave-0n27"
server_key: "f2875bf2c1f829ec21158c968c784c33"
server_key_iv: "2914c50ef8686505abb8a7f05d128332"
server_key_version: "encryptionkey@0"
I20260928 14:44:59.617470 22585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:44:59.649371 22585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:44:59.652190 22585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:44:59.652395 22585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:44:59.659365 22585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40701
I20260928 14:44:59.659389 22652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40701 every 8 connection(s)
I20260928 14:44:59.660282 22585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:44:59.662911 22653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:44:59.664577 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 22585
I20260928 14:44:59.664726 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:44:59.665019 1828 external_mini_cluster.cc:1428] Setting key d8ad71d8ebd203c60b3fa6bca6526619
I20260928 14:44:59.668973 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b: Bootstrap starting.
I20260928 14:44:59.671211 22653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b: Neither blocks nor log segments found. Creating new log.
I20260928 14:44:59.672017 22653 log.cc:824] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:44:59.673763 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:59.666400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33995 (local address 127.0.0.1:40701)
0928 14:44:59.666851 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:59.666866 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:59.666900 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:44:59.667522 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:59.667538 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:59.667885 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:59.668186 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:59.668573 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:59.669631 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:59.670576 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:59.671060 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:59.671239 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:59.672007 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:59.672012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:59.672023 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:59.672504 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:59.672563 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:59.672592 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:59.672700 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:59.672880 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 14:44:59.673088 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":158,"threads_started":1}
I20260928 14:44:59.674014 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b: No bootstrap required, opened a new log
I20260928 14:44:59.677034 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:44:59.677860 22653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5897fd9494a6995537dfb5aabcb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40701 } }
I20260928 14:44:59.678054 22653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:44:59.678109 22653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc5897fd9494a6995537dfb5aabcb5b, State: Initialized, Role: FOLLOWER
I20260928 14:44:59.678565 22653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5897fd9494a6995537dfb5aabcb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40701 } }
I20260928 14:44:59.678720 22653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:44:59.678819 22653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:44:59.678927 22653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:44:59.679849 22653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5897fd9494a6995537dfb5aabcb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40701 } }
I20260928 14:44:59.680171 22653 leader_election.cc:302] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dc5897fd9494a6995537dfb5aabcb5b; no voters:
I20260928 14:44:59.680495 22653 leader_election.cc:288] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:44:59.680558 22663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:44:59.680764 22663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [term 1 LEADER]: Becoming Leader. State: Replica: 9dc5897fd9494a6995537dfb5aabcb5b, State: Running, Role: LEADER
I20260928 14:44:59.681088 22663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5897fd9494a6995537dfb5aabcb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40701 } }
I20260928 14:44:59.681706 22653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:44:59.682801 22664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dc5897fd9494a6995537dfb5aabcb5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5897fd9494a6995537dfb5aabcb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40701 } } }
I20260928 14:44:59.682972 22664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [sys.catalog]: This master's current role is: LEADER
I20260928 14:44:59.683221 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:44:59.679081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47961 (local address 127.0.0.1:40701)
0928 14:44:59.679202 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 14:44:59.679207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:44:59.679364 (+ 157us) server_negotiation.cc:407] Connection header received
0928 14:44:59.679480 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:44:59.679483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:44:59.679520 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:44:59.679610 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:44:59.679861 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:59.680471 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:59.681498 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:44:59.681723 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:44:59.681884 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:44:59.682744 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:44:59.682748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:44:59.682750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:44:59.682762 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:44:59.682789 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:44:59.682801 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:44:59.682847 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:44:59.683029 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 14:44:59.683079 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:44:59.683538 22665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dc5897fd9494a6995537dfb5aabcb5b. Latest consensus state: current_term: 1 leader_uuid: "9dc5897fd9494a6995537dfb5aabcb5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5897fd9494a6995537dfb5aabcb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40701 } } }
I20260928 14:44:59.684558 22665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b [sys.catalog]: This master's current role is: LEADER
W20260928 14:44:59.686805 22678 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:44:59.686866 22678 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:44:59.686964 22672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:44:59.687605 22672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:44:59.693073 22672 catalog_manager.cc:1347] Generated new cluster ID: 039e7d12799b44c690c7418c308cd50f
I20260928 14:44:59.693147 22672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:44:59.706549 22672 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:44:59.707981 22672 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:44:59.717027 22672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dc5897fd9494a6995537dfb5aabcb5b: Generated new TSK 0
I20260928 14:44:59.717868 22672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:44:59.725076 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 22585
2026-09-28T14:44:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:44:59 dist-test-slave-0n27 krb5kdc[22684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:44:59 dist-test-slave-0n27 krb5kdc[22684](info): set up 2 sockets
Sep 28 14:44:59 dist-test-slave-0n27 krb5kdc[22684](info): commencing operation
krb5kdc: starting...
W20260928 14:45:00.746793 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:00 dist-test-slave-0n27 krb5kdc[22684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:45:00.860903 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:00.967437 22697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:00.967717 22697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:00.967792 22697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:00.971720 22697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:00.971807 22697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:00.971844 22697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:00.971903 22697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:00.971941 22697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:00.976823 22697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:00.977826 22697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:00.978807 22697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:00.987196 22705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:00.987329 22702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:00.987195 22703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:00.987397 22697 server_base.cc:1034] running on GCE node
I20260928 14:45:00.988022 22697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:00.988958 22697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:00.990137 22697 hybrid_clock.cc:648] HybridClock initialized: now 1790606700990106 us; error 45 us; skew 500 ppm
Sep 28 14:45:00 dist-test-slave-0n27 krb5kdc[22684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:00.992686 22697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:45:00.993530 22697 webserver.cc:460] Webserver started at http://127.0.0.1:39243/ using document root <none> and password file <none>
I20260928 14:45:00.994118 22697 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:00.994211 22697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:00.994433 22697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:00.996402 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "6396d38faa144185b88b697139e84e25"
format_stamp: "Formatted at 2026-09-28 14:45:00 on dist-test-slave-0n27"
server_key: "e83dc7dfd8de3e73f47204100276efb0"
server_key_iv: "c156f5047f8ef74d7e38efb0f9d36439"
server_key_version: "encryptionkey@0"
I20260928 14:45:00.996966 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "6396d38faa144185b88b697139e84e25"
format_stamp: "Formatted at 2026-09-28 14:45:00 on dist-test-slave-0n27"
server_key: "e83dc7dfd8de3e73f47204100276efb0"
server_key_iv: "c156f5047f8ef74d7e38efb0f9d36439"
server_key_version: "encryptionkey@0"
I20260928 14:45:01.000475 22697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 14:45:01.002753 22712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:01.003841 22697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:45:01.003975 22697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "6396d38faa144185b88b697139e84e25"
format_stamp: "Formatted at 2026-09-28 14:45:00 on dist-test-slave-0n27"
server_key: "e83dc7dfd8de3e73f47204100276efb0"
server_key_iv: "c156f5047f8ef74d7e38efb0f9d36439"
server_key_version: "encryptionkey@0"
I20260928 14:45:01.004082 22697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:01.008253 22697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:01.011018 22697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:01.011214 22697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:01.018337 22697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41827
I20260928 14:45:01.018353 22764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41827 every 8 connection(s)
I20260928 14:45:01.019241 22697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:01.021772 22765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:01.026057 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 22697
I20260928 14:45:01.026208 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:01.026497 1828 external_mini_cluster.cc:1428] Setting key c217edf5f2f41459de582e3a285cc59a
I20260928 14:45:01.027117 22765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25: Bootstrap starting.
I20260928 14:45:01.029315 22765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:01.030192 22765 log.cc:824] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:01.031366 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:01.027765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52601 (local address 127.0.0.1:41827)
0928 14:45:01.028151 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:01.028164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:01.028196 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:45:01.028800 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:01.028816 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:01.029076 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:01.029419 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:01.030226 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:01.030237 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:01.030250 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:01.030261 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:01.030345 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:01.030373 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:01.030475 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:01.030583 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 14:45:01.030748 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
I20260928 14:45:01.032044 22765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25: No bootstrap required, opened a new log
I20260928 14:45:01.034035 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:01.036391 22765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6396d38faa144185b88b697139e84e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } }
I20260928 14:45:01.036643 22765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:01.036702 22765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6396d38faa144185b88b697139e84e25, State: Initialized, Role: FOLLOWER
I20260928 14:45:01.037153 22765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6396d38faa144185b88b697139e84e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } }
I20260928 14:45:01.037315 22765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:01.037377 22765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:01.037474 22765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:01.038043 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:01.036326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46035 (local address 127.0.0.1:41827)
0928 14:45:01.036441 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:01.036445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:01.036735 (+ 290us) server_negotiation.cc:407] Connection header received
0928 14:45:01.036808 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:01.036810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:01.036840 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:01.036901 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:01.037650 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:01.037653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:01.037655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:01.037666 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:01.037690 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:01.037700 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:01.037762 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:01.037883 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 14:45:01.037919 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:45:01.038573 22765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6396d38faa144185b88b697139e84e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } }
I20260928 14:45:01.038928 22765 leader_election.cc:302] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6396d38faa144185b88b697139e84e25; no voters:
I20260928 14:45:01.039209 22765 leader_election.cc:288] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:01.039310 22775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:01.039527 22775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [term 1 LEADER]: Becoming Leader. State: Replica: 6396d38faa144185b88b697139e84e25, State: Running, Role: LEADER
I20260928 14:45:01.039881 22775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6396d38faa144185b88b697139e84e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } }
I20260928 14:45:01.040470 22765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:01.041682 22776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6396d38faa144185b88b697139e84e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6396d38faa144185b88b697139e84e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } } }
I20260928 14:45:01.041818 22776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:01.042110 22777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6396d38faa144185b88b697139e84e25. Latest consensus state: current_term: 1 leader_uuid: "6396d38faa144185b88b697139e84e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6396d38faa144185b88b697139e84e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } } }
I20260928 14:45:01.042212 22777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25 [sys.catalog]: This master's current role is: LEADER
W20260928 14:45:01.044855 22791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:45:01.044925 22791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:01.045001 22784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:01.045660 22784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:01.050751 22784 catalog_manager.cc:1347] Generated new cluster ID: 5d452cfcb68343e5854b723b6e3540b4
I20260928 14:45:01.050809 22784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:01.071239 22784 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:01.071551 22784 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:01.076928 22784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6396d38faa144185b88b697139e84e25: Generated new TSK 0
I20260928 14:45:01.077620 22784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:01.087920 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 22697
2026-09-28T14:45:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:01 dist-test-slave-0n27 krb5kdc[22796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:01 dist-test-slave-0n27 krb5kdc[22796](info): set up 2 sockets
Sep 28 14:45:01 dist-test-slave-0n27 krb5kdc[22796](info): commencing operation
krb5kdc: starting...
W20260928 14:45:02.098695 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:02 dist-test-slave-0n27 krb5kdc[22796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:45:02.214773 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:02.321554 22809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:02.321928 22809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:02.321972 22809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:02.325766 22809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:02.325874 22809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:02.325894 22809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:02.325911 22809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:02.325930 22809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:02.330928 22809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:02.331817 22809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:02.332716 22809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:02.341471 22815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:02.341455 22814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:02.341512 22809 server_base.cc:1034] running on GCE node
W20260928 14:45:02.341455 22817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:02.342381 22809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:02.343525 22809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:02.344686 22809 hybrid_clock.cc:648] HybridClock initialized: now 1790606702344670 us; error 28 us; skew 500 ppm
Sep 28 14:45:02 dist-test-slave-0n27 krb5kdc[22796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:02.347482 22809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:45:02.348300 22809 webserver.cc:460] Webserver started at http://127.0.0.1:43695/ using document root <none> and password file <none>
I20260928 14:45:02.348843 22809 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:02.348901 22809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:02.349104 22809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:02.351148 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "026f6373b72246cf94c84129beabbbd2"
format_stamp: "Formatted at 2026-09-28 14:45:02 on dist-test-slave-0n27"
server_key: "90b2b0a69970f226e717fcb138492e83"
server_key_iv: "ea94d7c81d04cf44b175156b44801f5b"
server_key_version: "encryptionkey@0"
I20260928 14:45:02.351699 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "026f6373b72246cf94c84129beabbbd2"
format_stamp: "Formatted at 2026-09-28 14:45:02 on dist-test-slave-0n27"
server_key: "90b2b0a69970f226e717fcb138492e83"
server_key_iv: "ea94d7c81d04cf44b175156b44801f5b"
server_key_version: "encryptionkey@0"
I20260928 14:45:02.355129 22809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 14:45:02.357363 22824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:02.358556 22809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:45:02.358685 22809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "026f6373b72246cf94c84129beabbbd2"
format_stamp: "Formatted at 2026-09-28 14:45:02 on dist-test-slave-0n27"
server_key: "90b2b0a69970f226e717fcb138492e83"
server_key_iv: "ea94d7c81d04cf44b175156b44801f5b"
server_key_version: "encryptionkey@0"
I20260928 14:45:02.358780 22809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:02.374871 22809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:02.378088 22809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:02.378300 22809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:02.385098 22809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35329
I20260928 14:45:02.385119 22876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35329 every 8 connection(s)
I20260928 14:45:02.386147 22809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:02.388759 22877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:02.390522 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 22809
I20260928 14:45:02.390671 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:02.390975 1828 external_mini_cluster.cc:1428] Setting key ba989a8cb35ad80ccd3dd69b126304a9
I20260928 14:45:02.394944 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2: Bootstrap starting.
I20260928 14:45:02.397106 22877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:02.397859 22877 log.cc:824] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:45:02 dist-test-slave-0n27 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:45:02.399955 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2: No bootstrap required, opened a new log
I20260928 14:45:02.404107 22877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026f6373b72246cf94c84129beabbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35329 } }
I20260928 14:45:02.404335 22877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:02.404412 22877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026f6373b72246cf94c84129beabbbd2, State: Initialized, Role: FOLLOWER
I20260928 14:45:02.404870 22877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026f6373b72246cf94c84129beabbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35329 } }
I20260928 14:45:02.404983 22877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:02.405066 22877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:02.405167 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:02.392317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:46959 (local address 127.0.0.1:35329)
0928 14:45:02.392716 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:02.392731 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:02.392762 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:45:02.393404 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:02.393419 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:02.393755 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:02.394157 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:02.394966 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:02.395995 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:02.396859 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:02.397413 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:02.397582 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:02.400504 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:02.400509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:02.400521 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:02.400545 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:02.402157 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:02.402691 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:02.402695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:02.402699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:02.402797 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:02.403117 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:02.403119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:02.403120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:02.403499 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:02.403663 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:02.403980 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 14:45:02.404231 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260928 14:45:02.405177 22877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:02.406119 22877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026f6373b72246cf94c84129beabbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35329 } }
I20260928 14:45:02.406498 22877 leader_election.cc:302] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 026f6373b72246cf94c84129beabbbd2; no voters:
I20260928 14:45:02.406817 22877 leader_election.cc:288] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:02.406883 22882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:02.407071 22882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [term 1 LEADER]: Becoming Leader. State: Replica: 026f6373b72246cf94c84129beabbbd2, State: Running, Role: LEADER
I20260928 14:45:02.407366 22882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026f6373b72246cf94c84129beabbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35329 } }
I20260928 14:45:02.408011 22877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:02.408327 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:02.408900 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 026f6373b72246cf94c84129beabbbd2. Latest consensus state: current_term: 1 leader_uuid: "026f6373b72246cf94c84129beabbbd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026f6373b72246cf94c84129beabbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35329 } } }
I20260928 14:45:02.409014 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:02.409364 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "026f6373b72246cf94c84129beabbbd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026f6373b72246cf94c84129beabbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35329 } } }
I20260928 14:45:02.409415 22891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:02.409446 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:02.412545 22891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:02.419159 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:02.410754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:56613 (local address 127.0.0.1:35329)
0928 14:45:02.410858 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:02.410862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:02.411399 (+ 537us) server_negotiation.cc:407] Connection header received
0928 14:45:02.411510 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:02.411513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:02.411559 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:02.411634 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:02.412412 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:02.413159 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:02.414196 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:02.414508 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:02.414610 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:02.415959 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:02.415963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:02.415968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:02.415994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:02.417462 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:02.417918 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:02.417923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:02.417927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:02.418010 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:02.418326 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:02.418329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:02.418331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:02.418506 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:02.418649 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:02.418906 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 14:45:02.418992 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:45:02.422130 22891 catalog_manager.cc:1347] Generated new cluster ID: bb98e5ae78f844a281c40e6f25520dbf
I20260928 14:45:02.422195 22891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:02.435806 22891 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:02.436659 22891 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:02.448303 22891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 026f6373b72246cf94c84129beabbbd2: Generated new TSK 0
I20260928 14:45:02.448922 22891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:02.452314 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 22809
W20260928 14:45:02.457700 22794 connection.cc:535] client connection to 127.0.0.1:35329 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35329): Connection reset by peer (error 104)
2026-09-28T14:45:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:02 dist-test-slave-0n27 krb5kdc[22908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:02 dist-test-slave-0n27 krb5kdc[22908](info): set up 2 sockets
Sep 28 14:45:02 dist-test-slave-0n27 krb5kdc[22908](info): commencing operation
krb5kdc: starting...
W20260928 14:45:03.467010 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:03 dist-test-slave-0n27 krb5kdc[22908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:45:03.586724 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:03.693495 22921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:03.693737 22921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:03.693823 22921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:03.697597 22921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:03.697645 22921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:03.697664 22921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:03.697723 22921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:03.697751 22921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:03.702838 22921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:03.703806 22921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:03.704792 22921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:03.713049 22926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:03.713073 22929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:03.713109 22927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:03.713261 22921 server_base.cc:1034] running on GCE node
I20260928 14:45:03.713876 22921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:03.714908 22921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:03.716110 22921 hybrid_clock.cc:648] HybridClock initialized: now 1790606703716101 us; error 28 us; skew 500 ppm
Sep 28 14:45:03 dist-test-slave-0n27 krb5kdc[22908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:03.723959 22921 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:45:03.724862 22921 webserver.cc:460] Webserver started at http://127.0.0.1:43289/ using document root <none> and password file <none>
I20260928 14:45:03.725409 22921 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:03.725499 22921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:03.725729 22921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:03.727757 22921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "8ae91971014d4102bb805a5a39110a7e"
format_stamp: "Formatted at 2026-09-28 14:45:03 on dist-test-slave-0n27"
server_key: "b4515fd8de28f285f5d45290043d673c"
server_key_iv: "bbe9425c24fc5036c969d782286d4d73"
server_key_version: "encryptionkey@0"
I20260928 14:45:03.728308 22921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "8ae91971014d4102bb805a5a39110a7e"
format_stamp: "Formatted at 2026-09-28 14:45:03 on dist-test-slave-0n27"
server_key: "b4515fd8de28f285f5d45290043d673c"
server_key_iv: "bbe9425c24fc5036c969d782286d4d73"
server_key_version: "encryptionkey@0"
I20260928 14:45:03.731989 22921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:45:03.734375 22936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:03.735544 22921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:45:03.735706 22921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "8ae91971014d4102bb805a5a39110a7e"
format_stamp: "Formatted at 2026-09-28 14:45:03 on dist-test-slave-0n27"
server_key: "b4515fd8de28f285f5d45290043d673c"
server_key_iv: "bbe9425c24fc5036c969d782286d4d73"
server_key_version: "encryptionkey@0"
I20260928 14:45:03.735934 22921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:03.746752 22921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:03.749428 22921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:03.749630 22921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:03.756192 22921 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33119
I20260928 14:45:03.756218 22988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33119 every 8 connection(s)
I20260928 14:45:03.757046 22921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:03.759604 22989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:03.762040 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 22921
I20260928 14:45:03.762202 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:03.762478 1828 external_mini_cluster.cc:1428] Setting key 9e7b75f2f402d8afdffe78ba2e174d16
I20260928 14:45:03.765051 22989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e: Bootstrap starting.
I20260928 14:45:03.767071 22989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:03.767767 22989 log.cc:824] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:03.769524 22989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e: No bootstrap required, opened a new log
I20260928 14:45:03.770815 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:03.763678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:53141 (local address 127.0.0.1:33119)
0928 14:45:03.764051 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:03.764066 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:03.764112 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:45:03.764668 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:03.764684 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:03.764986 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:03.765286 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:03.765652 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:03.766573 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:03.767499 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:03.767989 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:03.768159 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:03.768959 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:03.768969 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:03.768982 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:03.769343 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:03.769397 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:03.769418 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:03.769558 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:03.769796 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 14:45:03.769931 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":123,"threads_started":1}
I20260928 14:45:03.773161 22989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae91971014d4102bb805a5a39110a7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } }
I20260928 14:45:03.773350 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:03.773408 22989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:03.773478 22989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae91971014d4102bb805a5a39110a7e, State: Initialized, Role: FOLLOWER
I20260928 14:45:03.773888 22989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae91971014d4102bb805a5a39110a7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } }
I20260928 14:45:03.773998 22989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:03.774071 22989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:03.774185 22989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:03.775008 22989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae91971014d4102bb805a5a39110a7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } }
I20260928 14:45:03.775336 22989 leader_election.cc:302] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ae91971014d4102bb805a5a39110a7e; no voters:
I20260928 14:45:03.775696 22989 leader_election.cc:288] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:03.775760 22999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:03.775969 22999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [term 1 LEADER]: Becoming Leader. State: Replica: 8ae91971014d4102bb805a5a39110a7e, State: Running, Role: LEADER
I20260928 14:45:03.776252 22999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae91971014d4102bb805a5a39110a7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } }
I20260928 14:45:03.776926 22989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:03.777454 22999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ae91971014d4102bb805a5a39110a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae91971014d4102bb805a5a39110a7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } } }
I20260928 14:45:03.777626 22999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:03.777844 23000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ae91971014d4102bb805a5a39110a7e. Latest consensus state: current_term: 1 leader_uuid: "8ae91971014d4102bb805a5a39110a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae91971014d4102bb805a5a39110a7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } } }
I20260928 14:45:03.777935 23000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:03.778085 23007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:03.780139 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:03.775406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:33197 (local address 127.0.0.1:33119)
0928 14:45:03.775535 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:03.775539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:03.775839 (+ 300us) server_negotiation.cc:407] Connection header received
0928 14:45:03.775996 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:03.775998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:03.776039 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:03.776107 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:03.776383 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:03.777033 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:03.778198 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:03.778506 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:03.778576 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:03.779680 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:03.779685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:03.779691 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:03.779709 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:03.779736 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:03.779749 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:03.779844 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:03.779993 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 14:45:03.780026 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:45:03.781540 23007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:03.786947 23007 catalog_manager.cc:1347] Generated new cluster ID: ed6c8c09cf114fc9b4e3eb53e8ab87f7
I20260928 14:45:03.787017 23007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:03.814558 23007 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:03.815833 23007 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:03.821770 23007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ae91971014d4102bb805a5a39110a7e: Generated new TSK 0
I20260928 14:45:03.822548 23007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:03.829018 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 22921
W20260928 14:45:03.834323 22906 connection.cc:535] client connection to 127.0.0.1:33119 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33119): Connection reset by peer (error 104)
2026-09-28T14:45:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:03 dist-test-slave-0n27 krb5kdc[23020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:03 dist-test-slave-0n27 krb5kdc[23020](info): set up 2 sockets
Sep 28 14:45:03 dist-test-slave-0n27 krb5kdc[23020](info): commencing operation
krb5kdc: starting...
W20260928 14:45:04.834811 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:04 dist-test-slave-0n27 krb5kdc[23020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:45:04.951078 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:05.055898 23033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:05.056205 23033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:05.056308 23033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:05.060334 23033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:05.060422 23033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:05.060461 23033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:05.060497 23033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:05.060532 23033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:05.065423 23033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:05.066407 23033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:05.067360 23033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:05.075459 23039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:05.075526 23033 server_base.cc:1034] running on GCE node
W20260928 14:45:05.075544 23038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:05.075459 23041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:05.076222 23033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:05.077239 23033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:05.078430 23033 hybrid_clock.cc:648] HybridClock initialized: now 1790606705078416 us; error 22 us; skew 500 ppm
Sep 28 14:45:05 dist-test-slave-0n27 krb5kdc[23020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:05.081115 23033 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:45:05.081938 23033 webserver.cc:460] Webserver started at http://127.0.0.1:42983/ using document root <none> and password file <none>
I20260928 14:45:05.082412 23033 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:05.082465 23033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:05.082612 23033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:05.084405 23033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "30540a88133b418aaf7f4eb549f781f2"
format_stamp: "Formatted at 2026-09-28 14:45:05 on dist-test-slave-0n27"
server_key: "3fdae7cd1239110b52a51151fbe28d18"
server_key_iv: "02031090b47396ca8dbb6ec70c05dc53"
server_key_version: "encryptionkey@0"
I20260928 14:45:05.084877 23033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "30540a88133b418aaf7f4eb549f781f2"
format_stamp: "Formatted at 2026-09-28 14:45:05 on dist-test-slave-0n27"
server_key: "3fdae7cd1239110b52a51151fbe28d18"
server_key_iv: "02031090b47396ca8dbb6ec70c05dc53"
server_key_version: "encryptionkey@0"
I20260928 14:45:05.088229 23033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:45:05.090443 23048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:05.091588 23033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:45:05.091719 23033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "30540a88133b418aaf7f4eb549f781f2"
format_stamp: "Formatted at 2026-09-28 14:45:05 on dist-test-slave-0n27"
server_key: "3fdae7cd1239110b52a51151fbe28d18"
server_key_iv: "02031090b47396ca8dbb6ec70c05dc53"
server_key_version: "encryptionkey@0"
I20260928 14:45:05.091811 23033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:05.096073 23033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:05.098814 23033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:05.098997 23033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:05.105129 23033 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43989
I20260928 14:45:05.105176 23100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43989 every 8 connection(s)
I20260928 14:45:05.106122 23033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:05.108870 23101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:05.114114 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2: Bootstrap starting.
I20260928 14:45:05.116361 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 23033
I20260928 14:45:05.116508 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:05.116521 23101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:05.116801 1828 external_mini_cluster.cc:1428] Setting key 15f0cde738133b21788f3b7bd1c8a732
I20260928 14:45:05.117350 23101 log.cc:824] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:05.119573 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2: No bootstrap required, opened a new log
I20260928 14:45:05.122048 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:05.117895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:45087 (local address 127.0.0.1:43989)
0928 14:45:05.118346 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:05.118364 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:05.118403 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:45:05.119193 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:05.119215 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:05.119563 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:05.119947 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:05.120670 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:05.120675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:05.120690 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:05.120702 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:05.120763 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:05.120799 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:05.120911 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:05.121000 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 14:45:05.121150 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
I20260928 14:45:05.123939 23101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30540a88133b418aaf7f4eb549f781f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } }
I20260928 14:45:05.124150 23101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:05.124233 23101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30540a88133b418aaf7f4eb549f781f2, State: Initialized, Role: FOLLOWER
I20260928 14:45:05.124688 23101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30540a88133b418aaf7f4eb549f781f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } }
I20260928 14:45:05.124792 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:05.124872 23101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:05.124929 23101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:05.125016 23101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:05.126187 23101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30540a88133b418aaf7f4eb549f781f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } }
I20260928 14:45:05.126557 23101 leader_election.cc:302] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30540a88133b418aaf7f4eb549f781f2; no voters:
I20260928 14:45:05.126889 23101 leader_election.cc:288] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:05.126999 23110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:05.127310 23110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [term 1 LEADER]: Becoming Leader. State: Replica: 30540a88133b418aaf7f4eb549f781f2, State: Running, Role: LEADER
I20260928 14:45:05.127681 23110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30540a88133b418aaf7f4eb549f781f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } }
I20260928 14:45:05.128249 23101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:05.129037 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:05.127032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:35929 (local address 127.0.0.1:43989)
0928 14:45:05.127136 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:05.127141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:05.127394 (+ 253us) server_negotiation.cc:407] Connection header received
0928 14:45:05.127604 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:05.127608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:05.127641 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:05.127725 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:05.128487 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:05.128490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:05.128492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:05.128504 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:05.128530 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:05.128543 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:05.128665 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:05.128913 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 14:45:05.128957 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:45:05.129994 23113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30540a88133b418aaf7f4eb549f781f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30540a88133b418aaf7f4eb549f781f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } } }
I20260928 14:45:05.129963 23112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30540a88133b418aaf7f4eb549f781f2. Latest consensus state: current_term: 1 leader_uuid: "30540a88133b418aaf7f4eb549f781f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30540a88133b418aaf7f4eb549f781f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } } }
I20260928 14:45:05.130100 23112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:05.130100 23113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:05.130525 23118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:05.133023 23118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:05.137975 23118 catalog_manager.cc:1347] Generated new cluster ID: 343e8aeb64da4af1bf9eaac377bd807e
I20260928 14:45:05.138031 23118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:05.148649 23118 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:05.148926 23118 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:05.159029 23118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30540a88133b418aaf7f4eb549f781f2: Generated new TSK 0
I20260928 14:45:05.159736 23118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:05.172407 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 23033
2026-09-28T14:45:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:05 dist-test-slave-0n27 krb5kdc[23132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:05 dist-test-slave-0n27 krb5kdc[23132](info): set up 2 sockets
Sep 28 14:45:05 dist-test-slave-0n27 krb5kdc[23132](info): commencing operation
krb5kdc: starting...
W20260928 14:45:06.186858 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:06 dist-test-slave-0n27 krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:45:06.304420 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42095
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:06.408104 23145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:06.408372 23145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:06.408456 23145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:06.412432 23145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:06.412511 23145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:06.412555 23145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:06.412611 23145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:06.412647 23145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:06.417691 23145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42095
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:06.418752 23145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:06.419761 23145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:06.427874 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:06.427853 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:06.427853 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:06.427958 23145 server_base.cc:1034] running on GCE node
I20260928 14:45:06.428663 23145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:06.429716 23145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:06.430948 23145 hybrid_clock.cc:648] HybridClock initialized: now 1790606706430905 us; error 45 us; skew 500 ppm
Sep 28 14:45:06 dist-test-slave-0n27 krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:06.433403 23145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:45:06.434273 23145 webserver.cc:460] Webserver started at http://127.0.0.1:43113/ using document root <none> and password file <none>
I20260928 14:45:06.434820 23145 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:06.434906 23145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:06.435130 23145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:06.437064 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "560daa3419f94ac0ab775b48bc2b44a7"
format_stamp: "Formatted at 2026-09-28 14:45:06 on dist-test-slave-0n27"
server_key: "a335fe7def683b96972f710b9b151fcc"
server_key_iv: "26b1a84d244214a9914cf8f7301cef16"
server_key_version: "encryptionkey@0"
I20260928 14:45:06.437615 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "560daa3419f94ac0ab775b48bc2b44a7"
format_stamp: "Formatted at 2026-09-28 14:45:06 on dist-test-slave-0n27"
server_key: "a335fe7def683b96972f710b9b151fcc"
server_key_iv: "26b1a84d244214a9914cf8f7301cef16"
server_key_version: "encryptionkey@0"
I20260928 14:45:06.441069 23145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:06.443405 23160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:06.444559 23145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:45:06.444792 23145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "560daa3419f94ac0ab775b48bc2b44a7"
format_stamp: "Formatted at 2026-09-28 14:45:06 on dist-test-slave-0n27"
server_key: "a335fe7def683b96972f710b9b151fcc"
server_key_iv: "26b1a84d244214a9914cf8f7301cef16"
server_key_version: "encryptionkey@0"
I20260928 14:45:06.444902 23145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:06.457176 23145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:06.459792 23145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:06.459983 23145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:06.466327 23145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42095
I20260928 14:45:06.466391 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42095 every 8 connection(s)
I20260928 14:45:06.467214 23145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:06.469540 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 23145
I20260928 14:45:06.469729 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:06.470052 23213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:06.470063 1828 external_mini_cluster.cc:1428] Setting key 891fd457c54211bcbd055b21b13f35e6
I20260928 14:45:06.476728 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7: Bootstrap starting.
Sep 28 14:45:06 dist-test-slave-0n27 krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:45:06.479393 23213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:06.480111 23213 log.cc:824] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:06.482358 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7: No bootstrap required, opened a new log
I20260928 14:45:06.484750 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:06.471353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:42367 (local address 127.0.0.1:42095)
0928 14:45:06.471756 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:06.471770 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:06.471802 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:45:06.472542 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:06.472558 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:06.472898 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:06.473156 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:06.474110 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:06.475044 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:06.476233 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:06.476635 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:06.476783 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:06.479512 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:06.479520 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:06.479536 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:06.479568 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:06.481559 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:06.482112 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:06.482118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:06.482125 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:06.482260 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:06.482620 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:06.482623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:06.482624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:06.483100 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:06.483299 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:06.483528 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 14:45:06.483770 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":135,"threads_started":1}
I20260928 14:45:06.486618 23213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560daa3419f94ac0ab775b48bc2b44a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42095 } }
I20260928 14:45:06.486868 23213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:06.486927 23213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560daa3419f94ac0ab775b48bc2b44a7, State: Initialized, Role: FOLLOWER
I20260928 14:45:06.487337 23213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560daa3419f94ac0ab775b48bc2b44a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42095 } }
I20260928 14:45:06.487488 23213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:06.487557 23213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:06.487663 23213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:06.487767 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:06.488744 23213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560daa3419f94ac0ab775b48bc2b44a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42095 } }
I20260928 14:45:06.489111 23213 leader_election.cc:302] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 560daa3419f94ac0ab775b48bc2b44a7; no voters:
I20260928 14:45:06.489434 23213 leader_election.cc:288] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:06.489527 23222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:06.489845 23222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [term 1 LEADER]: Becoming Leader. State: Replica: 560daa3419f94ac0ab775b48bc2b44a7, State: Running, Role: LEADER
I20260928 14:45:06.490212 23222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560daa3419f94ac0ab775b48bc2b44a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42095 } }
I20260928 14:45:06.490658 23213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:06.491261 23222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 560daa3419f94ac0ab775b48bc2b44a7. Latest consensus state: current_term: 1 leader_uuid: "560daa3419f94ac0ab775b48bc2b44a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560daa3419f94ac0ab775b48bc2b44a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42095 } } }
I20260928 14:45:06.491420 23222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:06.491731 23223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "560daa3419f94ac0ab775b48bc2b44a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560daa3419f94ac0ab775b48bc2b44a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42095 } } }
I20260928 14:45:06.491849 23232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:06.491883 23223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:06.495447 23232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:06.501297 23232 catalog_manager.cc:1347] Generated new cluster ID: 99f7d6ba190f42ad87bb1e687c5d78b6
I20260928 14:45:06.501350 23232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:06.501505 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:06.489929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:54979 (local address 127.0.0.1:42095)
0928 14:45:06.490022 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:06.490025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:06.492736 (+ 2711us) server_negotiation.cc:407] Connection header received
0928 14:45:06.492841 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:06.492844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:06.492889 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:06.492968 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:06.493743 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:06.494583 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:06.495776 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:06.496138 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:06.496204 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:06.497847 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:06.497851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:06.497854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:06.497879 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:06.499583 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:06.500145 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:06.500148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:06.500150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:06.500233 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:06.500553 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:06.500556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:06.500557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:06.500736 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:06.500911 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:06.501221 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 14:45:06.501315 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:45:06.530966 23232 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:06.531809 23232 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:06.535923 23232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 560daa3419f94ac0ab775b48bc2b44a7: Generated new TSK 0
I20260928 14:45:06.536552 23232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:06.543073 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 23145
W20260928 14:45:06.548324 23130 connection.cc:535] client connection to 127.0.0.1:42095 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42095): Connection reset by peer (error 104)
2026-09-28T14:45:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:06 dist-test-slave-0n27 krb5kdc[23244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:06 dist-test-slave-0n27 krb5kdc[23244](info): set up 2 sockets
Sep 28 14:45:06 dist-test-slave-0n27 krb5kdc[23244](info): commencing operation
krb5kdc: starting...
W20260928 14:45:07.538475 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:07 dist-test-slave-0n27 krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:45:07.656086 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:07.762233 23257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:07.762497 23257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:07.762590 23257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:07.766294 23257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:07.766362 23257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:07.766395 23257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:07.766467 23257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:07.766503 23257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:07.771492 23257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35293
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:07.772457 23257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:07.773429 23257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:07.781390 23265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:07.781579 23263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:07.781632 23257 server_base.cc:1034] running on GCE node
W20260928 14:45:07.781536 23262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:07.782313 23257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:07.783358 23257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:07.784541 23257 hybrid_clock.cc:648] HybridClock initialized: now 1790606707784524 us; error 28 us; skew 500 ppm
Sep 28 14:45:07 dist-test-slave-0n27 krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:07.787201 23257 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:45:07.788048 23257 webserver.cc:460] Webserver started at http://127.0.0.1:34511/ using document root <none> and password file <none>
I20260928 14:45:07.788620 23257 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:07.788712 23257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:07.788933 23257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:07.791065 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e"
format_stamp: "Formatted at 2026-09-28 14:45:07 on dist-test-slave-0n27"
server_key: "4dce2871640b3b07bc5aaa3afba1703b"
server_key_iv: "a325748ed508905050b9c07517eb6443"
server_key_version: "encryptionkey@0"
I20260928 14:45:07.791627 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e"
format_stamp: "Formatted at 2026-09-28 14:45:07 on dist-test-slave-0n27"
server_key: "4dce2871640b3b07bc5aaa3afba1703b"
server_key_iv: "a325748ed508905050b9c07517eb6443"
server_key_version: "encryptionkey@0"
I20260928 14:45:07.795198 23257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:45:07.797483 23272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:07.798763 23257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:45:07.798913 23257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e"
format_stamp: "Formatted at 2026-09-28 14:45:07 on dist-test-slave-0n27"
server_key: "4dce2871640b3b07bc5aaa3afba1703b"
server_key_iv: "a325748ed508905050b9c07517eb6443"
server_key_version: "encryptionkey@0"
I20260928 14:45:07.799022 23257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:07.822597 23257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:07.825719 23257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:07.825928 23257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:07.833113 23257 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35293
I20260928 14:45:07.833144 23324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35293 every 8 connection(s)
I20260928 14:45:07.834050 23257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:07.836714 23325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:07.841940 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 23257
I20260928 14:45:07.842041 23325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e: Bootstrap starting.
I20260928 14:45:07.842080 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:07.842355 1828 external_mini_cluster.cc:1428] Setting key 67e4025b4e21112d96708010d18b5a11
I20260928 14:45:07.844496 23325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:07.845150 23325 log.cc:824] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:07.846938 23325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e: No bootstrap required, opened a new log
I20260928 14:45:07.850237 23325 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260928 14:45:07.850476 23325 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:07.850551 23325 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc78dda3e5a43fb8f44e0dedf819f3e, State: Initialized, Role: FOLLOWER
I20260928 14:45:07.850950 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:07.843639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:48735 (local address 127.0.0.1:35293)
0928 14:45:07.844064 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:07.844078 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:07.844124 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:45:07.844789 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:07.844805 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:07.845134 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:07.845489 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:07.845929 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:07.846841 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:07.847706 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:07.848144 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:07.848299 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:07.849209 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:07.849214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:07.849226 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:07.849578 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:07.849632 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:07.849654 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:07.849791 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:07.850018 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 14:45:07.850155 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260928 14:45:07.851044 23325 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260928 14:45:07.851220 23325 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:07.851287 23325 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:07.851379 23325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:07.852290 23325 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260928 14:45:07.852643 23325 leader_election.cc:302] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddc78dda3e5a43fb8f44e0dedf819f3e; no voters:
I20260928 14:45:07.852957 23325 leader_election.cc:288] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:07.853039 23330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:07.853243 23330 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [term 1 LEADER]: Becoming Leader. State: Replica: ddc78dda3e5a43fb8f44e0dedf819f3e, State: Running, Role: LEADER
I20260928 14:45:07.853614 23330 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260928 14:45:07.853875 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:07.854188 23325 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:07.855257 23331 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddc78dda3e5a43fb8f44e0dedf819f3e. Latest consensus state: current_term: 1 leader_uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } } }
I20260928 14:45:07.855362 23331 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:07.855635 23332 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc78dda3e5a43fb8f44e0dedf819f3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } } }
I20260928 14:45:07.855733 23332 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:07.855722 23343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:07.859164 23343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:07.860531 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:07.856126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:53059 (local address 127.0.0.1:35293)
0928 14:45:07.856298 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:07.856304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:07.856872 (+ 568us) server_negotiation.cc:407] Connection header received
0928 14:45:07.856978 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:07.856981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:07.857022 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:07.857095 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:07.857394 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:07.858070 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:07.858871 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:07.859126 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:07.859237 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:07.860051 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:07.860054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:07.860057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:07.860069 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:07.860102 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:07.860113 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:07.860218 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:07.860364 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 14:45:07.860397 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 14:45:07.864585 23343 catalog_manager.cc:1347] Generated new cluster ID: 479d7742d596440283b8d677ae3f123a
I20260928 14:45:07.864645 23343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:07.902710 23343 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:07.904057 23343 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:07.911381 23343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddc78dda3e5a43fb8f44e0dedf819f3e: Generated new TSK 0
I20260928 14:45:07.912209 23343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:07.923884 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 23257
W20260928 14:45:07.929365 23242 connection.cc:535] client connection to 127.0.0.1:35293 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35293): Connection reset by peer (error 104)
2026-09-28T14:45:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:07 dist-test-slave-0n27 krb5kdc[23356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:07 dist-test-slave-0n27 krb5kdc[23356](info): set up 2 sockets
Sep 28 14:45:07 dist-test-slave-0n27 krb5kdc[23356](info): commencing operation
krb5kdc: starting...
W20260928 14:45:08.922935 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:09 dist-test-slave-0n27 krb5kdc[23356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:45:09.040314 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:09.148236 23369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:09.148499 23369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:09.148562 23369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:09.152503 23369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:09.152582 23369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:09.152619 23369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:09.152678 23369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:09.152717 23369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:09.157630 23369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:09.158635 23369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:09.159603 23369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:09.167984 23374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:09.167979 23375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:09.168036 23369 server_base.cc:1034] running on GCE node
W20260928 14:45:09.167984 23377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:09.168716 23369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:09.169704 23369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:09.170861 23369 hybrid_clock.cc:648] HybridClock initialized: now 1790606709170845 us; error 28 us; skew 500 ppm
Sep 28 14:45:09 dist-test-slave-0n27 krb5kdc[23356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:09.181272 23369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:45:09.182199 23369 webserver.cc:460] Webserver started at http://127.0.0.1:38225/ using document root <none> and password file <none>
I20260928 14:45:09.182934 23369 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:09.183032 23369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:09.183298 23369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:09.185236 23369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "0fb5c665131346edb1f15e4d1ab682f2"
format_stamp: "Formatted at 2026-09-28 14:45:09 on dist-test-slave-0n27"
server_key: "edeed079976fc1442bfb408b6f2fbf41"
server_key_iv: "1fa1e99afcd3f3d16017963a5a81997e"
server_key_version: "encryptionkey@0"
I20260928 14:45:09.185865 23369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "0fb5c665131346edb1f15e4d1ab682f2"
format_stamp: "Formatted at 2026-09-28 14:45:09 on dist-test-slave-0n27"
server_key: "edeed079976fc1442bfb408b6f2fbf41"
server_key_iv: "1fa1e99afcd3f3d16017963a5a81997e"
server_key_version: "encryptionkey@0"
I20260928 14:45:09.189324 23369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:45:09.191644 23384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:09.192803 23369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:45:09.192898 23369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "0fb5c665131346edb1f15e4d1ab682f2"
format_stamp: "Formatted at 2026-09-28 14:45:09 on dist-test-slave-0n27"
server_key: "edeed079976fc1442bfb408b6f2fbf41"
server_key_iv: "1fa1e99afcd3f3d16017963a5a81997e"
server_key_version: "encryptionkey@0"
I20260928 14:45:09.192972 23369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:09.197431 23369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:09.200162 23369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:09.200309 23369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:09.206817 23369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33117
I20260928 14:45:09.206843 23436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33117 every 8 connection(s)
I20260928 14:45:09.207675 23369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:09.210425 23437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:09.215623 23437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2: Bootstrap starting.
I20260928 14:45:09.216060 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 23369
I20260928 14:45:09.216202 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:09.216475 1828 external_mini_cluster.cc:1428] Setting key c7c4fa53bd45eb6e01d16aa14505956b
I20260928 14:45:09.218242 23437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:09.218876 23437 log.cc:824] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:09.221020 23437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2: No bootstrap required, opened a new log
I20260928 14:45:09.221586 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:09.217715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:50111 (local address 127.0.0.1:33117)
0928 14:45:09.218138 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:09.218150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:09.218192 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:45:09.218856 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:09.218873 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:09.219216 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:09.219570 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:09.220369 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:09.220374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:09.220385 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:09.220394 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:09.220443 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:09.220460 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:09.220543 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:09.220697 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 14:45:09.220872 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":104,"threads_started":1}
I20260928 14:45:09.224318 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:09.224874 23437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb5c665131346edb1f15e4d1ab682f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260928 14:45:09.225158 23437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:09.225219 23437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb5c665131346edb1f15e4d1ab682f2, State: Initialized, Role: FOLLOWER
I20260928 14:45:09.225683 23437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb5c665131346edb1f15e4d1ab682f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260928 14:45:09.225889 23437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:09.225960 23437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:09.226053 23437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:09.227133 23437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb5c665131346edb1f15e4d1ab682f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260928 14:45:09.227474 23437 leader_election.cc:302] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fb5c665131346edb1f15e4d1ab682f2; no voters:
I20260928 14:45:09.227823 23437 leader_election.cc:288] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:09.227909 23447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:09.228111 23447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [term 1 LEADER]: Becoming Leader. State: Replica: 0fb5c665131346edb1f15e4d1ab682f2, State: Running, Role: LEADER
I20260928 14:45:09.228152 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:09.226514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:57551 (local address 127.0.0.1:33117)
0928 14:45:09.226631 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:09.226635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:09.226798 (+ 163us) server_negotiation.cc:407] Connection header received
0928 14:45:09.226914 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:09.226915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:09.226941 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:09.226999 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:09.227689 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:09.227692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:09.227694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:09.227704 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:09.227725 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:09.227734 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:09.227841 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:09.227991 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 14:45:09.228025 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:45:09.228622 23447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb5c665131346edb1f15e4d1ab682f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260928 14:45:09.229019 23437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:09.229844 23447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fb5c665131346edb1f15e4d1ab682f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb5c665131346edb1f15e4d1ab682f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } } }
I20260928 14:45:09.230039 23447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:09.230258 23447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fb5c665131346edb1f15e4d1ab682f2. Latest consensus state: current_term: 1 leader_uuid: "0fb5c665131346edb1f15e4d1ab682f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb5c665131346edb1f15e4d1ab682f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } } }
I20260928 14:45:09.230394 23447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:09.230402 23456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:09.233265 23456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:09.238651 23456 catalog_manager.cc:1347] Generated new cluster ID: 03816f5abb9448cbbd63a0c018ebcce7
I20260928 14:45:09.238719 23456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:09.251677 23456 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:09.251981 23456 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:09.267345 23456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fb5c665131346edb1f15e4d1ab682f2: Generated new TSK 0
I20260928 14:45:09.268203 23456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:09.271236 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 23369
2026-09-28T14:45:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:09 dist-test-slave-0n27 krb5kdc[23468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:09 dist-test-slave-0n27 krb5kdc[23468](info): set up 2 sockets
Sep 28 14:45:09 dist-test-slave-0n27 krb5kdc[23468](info): commencing operation
krb5kdc: starting...
W20260928 14:45:10.290781 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:10 dist-test-slave-0n27 krb5kdc[23468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:45:10.406922 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42571
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:10.513518 23481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:10.513741 23481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:10.513818 23481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:10.517403 23481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:10.517457 23481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:10.517475 23481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:10.517550 23481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:10.517583 23481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:10.522661 23481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:42571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:10.523615 23481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:10.524540 23481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:10.532853 23489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:10.532928 23481 server_base.cc:1034] running on GCE node
W20260928 14:45:10.532888 23487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:10.532853 23486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:10.533655 23481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:10.534683 23481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:10.535826 23481 hybrid_clock.cc:648] HybridClock initialized: now 1790606710535814 us; error 23 us; skew 500 ppm
Sep 28 14:45:10 dist-test-slave-0n27 krb5kdc[23468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606710, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:10.538367 23481 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:45:10.539258 23481 webserver.cc:460] Webserver started at http://127.1.201.62:34663/ using document root <none> and password file <none>
I20260928 14:45:10.539763 23481 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:10.539817 23481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:10.539968 23481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:10.541927 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "de542756fe944cffaf525b925ca8ded1"
format_stamp: "Formatted at 2026-09-28 14:45:10 on dist-test-slave-0n27"
server_key: "00d21764f678212c009ecae27e0a3d69"
server_key_iv: "4e56a96a681c35e65f71e9cdab5e6bc1"
server_key_version: "encryptionkey@0"
I20260928 14:45:10.542425 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "de542756fe944cffaf525b925ca8ded1"
format_stamp: "Formatted at 2026-09-28 14:45:10 on dist-test-slave-0n27"
server_key: "00d21764f678212c009ecae27e0a3d69"
server_key_iv: "4e56a96a681c35e65f71e9cdab5e6bc1"
server_key_version: "encryptionkey@0"
I20260928 14:45:10.545763 23481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:45:10.547989 23496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:10.549160 23481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:45:10.549292 23481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "de542756fe944cffaf525b925ca8ded1"
format_stamp: "Formatted at 2026-09-28 14:45:10 on dist-test-slave-0n27"
server_key: "00d21764f678212c009ecae27e0a3d69"
server_key_iv: "4e56a96a681c35e65f71e9cdab5e6bc1"
server_key_version: "encryptionkey@0"
I20260928 14:45:10.549389 23481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:10.564572 23481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:10.567696 23481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:10.567891 23481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:10.574548 23481 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:42571
I20260928 14:45:10.574625 23548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:42571 every 8 connection(s)
I20260928 14:45:10.575476 23481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:10.578164 23549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:10.582890 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 23481
I20260928 14:45:10.583029 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:10.583277 1828 external_mini_cluster.cc:1428] Setting key 2af83d4edc520b062ab4e0c854201743
I20260928 14:45:10.583678 23549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1: Bootstrap starting.
I20260928 14:45:10.586323 23549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:10.587025 23549 log.cc:824] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:10.588768 23549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1: No bootstrap required, opened a new log
Sep 28 14:45:10 dist-test-slave-0n27 krb5kdc[23468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:45:10.592303 23549 raft_consensus.cc:348] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de542756fe944cffaf525b925ca8ded1" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42571 } }
I20260928 14:45:10.592530 23549 raft_consensus.cc:374] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:10.592614 23549 raft_consensus.cc:729] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de542756fe944cffaf525b925ca8ded1, State: Initialized, Role: FOLLOWER
I20260928 14:45:10.593130 23549 consensus_queue.cc:260] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de542756fe944cffaf525b925ca8ded1" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42571 } }
I20260928 14:45:10.593299 23549 raft_consensus.cc:388] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:10.593369 23549 raft_consensus.cc:482] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:10.593483 23549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:10.594378 23549 raft_consensus.cc:504] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de542756fe944cffaf525b925ca8ded1" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42571 } }
I20260928 14:45:10.594750 23549 leader_election.cc:302] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de542756fe944cffaf525b925ca8ded1; no voters:
I20260928 14:45:10.595089 23549 leader_election.cc:288] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:10.595177 23554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:10.595391 23554 raft_consensus.cc:686] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [term 1 LEADER]: Becoming Leader. State: Replica: de542756fe944cffaf525b925ca8ded1, State: Running, Role: LEADER
I20260928 14:45:10.595770 23554 consensus_queue.cc:237] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de542756fe944cffaf525b925ca8ded1" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42571 } }
I20260928 14:45:10.596342 23549 sys_catalog.cc:564] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:10.596987 23554 sys_catalog.cc:455] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de542756fe944cffaf525b925ca8ded1. Latest consensus state: current_term: 1 leader_uuid: "de542756fe944cffaf525b925ca8ded1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de542756fe944cffaf525b925ca8ded1" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42571 } } }
I20260928 14:45:10.597396 23555 sys_catalog.cc:455] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de542756fe944cffaf525b925ca8ded1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de542756fe944cffaf525b925ca8ded1" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 42571 } } }
I20260928 14:45:10.597646 23555 sys_catalog.cc:458] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:10.597872 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:10.584594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37729 (local address 127.1.201.62:42571)
0928 14:45:10.585260 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:10.585279 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:10.585313 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:45:10.585983 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:10.586001 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:10.586326 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:10.586679 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:10.587481 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:10.588417 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:10.589325 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:10.589848 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:10.590016 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:10.592449 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:10.592454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:10.592466 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:10.592492 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:10.594358 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:10.594950 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:10.594954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:10.594958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:10.595095 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:10.595533 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:10.595536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:10.595538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:10.596061 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:10.596263 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:10.596664 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 14:45:10.596901 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":156,"threads_started":1}
I20260928 14:45:10.597177 23554 sys_catalog.cc:458] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:10.600999 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 14:45:10.601714 23569 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:45:10.601806 23569 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:10.601872 23563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:10.602753 23563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:10.608299 23563 catalog_manager.cc:1347] Generated new cluster ID: 0bed42012382498eb0adee027929777d
I20260928 14:45:10.608353 23563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:10.614319 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:10.604379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46771 (local address 127.1.201.62:42571)
0928 14:45:10.604502 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:10.604507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:10.604519 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:45:10.604556 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:10.604559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:10.604602 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:10.604680 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:10.605288 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:10.606231 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:10.607275 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:10.607646 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:10.607763 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:10.610887 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:10.610891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:10.610894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:10.610919 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:10.612577 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:10.613071 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:10.613074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:10.613076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:10.613156 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:10.613466 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:10.613469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:10.613471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:10.613670 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:10.613824 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:10.614075 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 14:45:10.614165 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:45:10.621346 23563 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:10.622258 23563 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:10.630774 23563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de542756fe944cffaf525b925ca8ded1: Generated new TSK 0
I20260928 14:45:10.631426 23563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:10.635927 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 23481
W20260928 14:45:10.641875 23466 connection.cc:535] client connection to 127.1.201.62:42571 recv error: Network error: failed to read from TLS socket (remote: 127.1.201.62:42571): Connection reset by peer (error 104)
2026-09-28T14:45:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:10 dist-test-slave-0n27 krb5kdc[23580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:10 dist-test-slave-0n27 krb5kdc[23580](info): set up 2 sockets
Sep 28 14:45:10 dist-test-slave-0n27 krb5kdc[23580](info): commencing operation
krb5kdc: starting...
W20260928 14:45:11.634915 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:11 dist-test-slave-0n27 krb5kdc[23580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:45:11.751904 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:38107
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:38107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:11.857155 23593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:11.857442 23593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:11.857517 23593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:11.861486 23593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:11.861567 23593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:11.861601 23593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:11.861634 23593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:11.861661 23593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:11.866595 23593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:38107
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:11.867558 23593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:11.868685 23593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:11.877130 23601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:11.877189 23593 server_base.cc:1034] running on GCE node
W20260928 14:45:11.877138 23598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:11.877126 23599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:11.877980 23593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:11.879400 23593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:11.880584 23593 hybrid_clock.cc:648] HybridClock initialized: now 1790606711880562 us; error 34 us; skew 500 ppm
Sep 28 14:45:11 dist-test-slave-0n27 krb5kdc[23580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606711, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:11.883244 23593 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:45:11.884057 23593 webserver.cc:460] Webserver started at http://127.1.201.62:34165/ using document root <none> and password file <none>
I20260928 14:45:11.884593 23593 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:11.884680 23593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:11.884903 23593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:11.886955 23593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "0c65c39ad4f744739a58afbebbdb6cea"
format_stamp: "Formatted at 2026-09-28 14:45:11 on dist-test-slave-0n27"
server_key: "905a22c9feb5be0343add1b23a480b26"
server_key_iv: "e1c8934bae84a874b9afcbffa6b36033"
server_key_version: "encryptionkey@0"
I20260928 14:45:11.887524 23593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "0c65c39ad4f744739a58afbebbdb6cea"
format_stamp: "Formatted at 2026-09-28 14:45:11 on dist-test-slave-0n27"
server_key: "905a22c9feb5be0343add1b23a480b26"
server_key_iv: "e1c8934bae84a874b9afcbffa6b36033"
server_key_version: "encryptionkey@0"
I20260928 14:45:11.891260 23593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:45:11.893551 23608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:11.894817 23593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:45:11.894965 23593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "0c65c39ad4f744739a58afbebbdb6cea"
format_stamp: "Formatted at 2026-09-28 14:45:11 on dist-test-slave-0n27"
server_key: "905a22c9feb5be0343add1b23a480b26"
server_key_iv: "e1c8934bae84a874b9afcbffa6b36033"
server_key_version: "encryptionkey@0"
I20260928 14:45:11.895087 23593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:11.922377 23593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:11.924999 23593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:11.925196 23593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:11.932024 23593 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:38107
I20260928 14:45:11.932045 23660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:38107 every 8 connection(s)
I20260928 14:45:11.932909 23593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:11.935540 23661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:11.937431 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 23593
I20260928 14:45:11.937572 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:11.937873 1828 external_mini_cluster.cc:1428] Setting key ba7008e3d49f94296987fb981062210c
I20260928 14:45:11.941869 23661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea: Bootstrap starting.
I20260928 14:45:11.944450 23661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:11.945240 23661 log.cc:824] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:11.946605 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:11.939149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42559 (local address 127.1.201.62:38107)
0928 14:45:11.939517 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:11.939541 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:11.939589 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:45:11.940212 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:11.940229 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:11.940545 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:11.940808 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:11.941143 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:11.942273 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:11.943452 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:11.943884 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:11.943995 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:11.944906 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:11.944911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:11.944922 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:11.945290 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:11.945340 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:11.945394 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:11.945514 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:11.945742 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 14:45:11.945890 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
I20260928 14:45:11.947178 23661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea: No bootstrap required, opened a new log
I20260928 14:45:11.949447 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:11.951323 23661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c65c39ad4f744739a58afbebbdb6cea" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38107 } }
I20260928 14:45:11.951614 23661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:11.951682 23661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c65c39ad4f744739a58afbebbdb6cea, State: Initialized, Role: FOLLOWER
I20260928 14:45:11.952160 23661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c65c39ad4f744739a58afbebbdb6cea" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38107 } }
I20260928 14:45:11.952363 23661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:11.952432 23661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:11.952527 23661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:11.953529 23661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c65c39ad4f744739a58afbebbdb6cea" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38107 } }
I20260928 14:45:11.953882 23661 leader_election.cc:302] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c65c39ad4f744739a58afbebbdb6cea; no voters:
I20260928 14:45:11.954258 23661 leader_election.cc:288] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:11.954319 23671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:11.954528 23671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [term 1 LEADER]: Becoming Leader. State: Replica: 0c65c39ad4f744739a58afbebbdb6cea, State: Running, Role: LEADER
I20260928 14:45:11.954855 23671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c65c39ad4f744739a58afbebbdb6cea" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38107 } }
I20260928 14:45:11.955466 23661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:11.955976 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:11.951479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39039 (local address 127.1.201.62:38107)
0928 14:45:11.951586 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:11.951590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:11.951725 (+ 135us) server_negotiation.cc:407] Connection header received
0928 14:45:11.951857 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:11.951859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:11.951911 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:11.951993 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:11.952252 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:11.952901 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:11.953972 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:11.954360 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:11.954505 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:11.955334 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:11.955337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:11.955340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:11.955352 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:11.955383 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:11.955393 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:11.955486 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:11.955813 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 14:45:11.955847 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:45:11.956099 23671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c65c39ad4f744739a58afbebbdb6cea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c65c39ad4f744739a58afbebbdb6cea" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38107 } } }
I20260928 14:45:11.956331 23671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:11.956619 23671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c65c39ad4f744739a58afbebbdb6cea. Latest consensus state: current_term: 1 leader_uuid: "0c65c39ad4f744739a58afbebbdb6cea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c65c39ad4f744739a58afbebbdb6cea" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 38107 } } }
I20260928 14:45:11.956773 23671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:11.957150 23680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:11.960232 23680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:11.965338 23680 catalog_manager.cc:1347] Generated new cluster ID: 1403b27ea73f471a8950cade90ba52e0
I20260928 14:45:11.965409 23680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:11.970353 23680 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:11.971462 23680 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:11.978746 23680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c65c39ad4f744739a58afbebbdb6cea: Generated new TSK 0
I20260928 14:45:11.979485 23680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:11.983227 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 23593
W20260928 14:45:11.987861 23578 connection.cc:535] client connection to 127.1.201.62:38107 recv error: Network error: failed to read from TLS socket (remote: 127.1.201.62:38107): Connection reset by peer (error 104)
2026-09-28T14:45:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:12 dist-test-slave-0n27 krb5kdc[23692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:12 dist-test-slave-0n27 krb5kdc[23692](info): set up 2 sockets
Sep 28 14:45:12 dist-test-slave-0n27 krb5kdc[23692](info): commencing operation
krb5kdc: starting...
W20260928 14:45:12.983014 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:13 dist-test-slave-0n27 krb5kdc[23692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:45:13.101064 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:36895
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:36895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:13.206908 23705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:13.207146 23705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:13.207186 23705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:13.210994 23705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:13.211042 23705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:13.211061 23705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:13.211077 23705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:13.211092 23705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:13.216259 23705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:36895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:36895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:13.217218 23705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:13.218225 23705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:13.226637 23711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:13.226676 23710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:13.226763 23705 server_base.cc:1034] running on GCE node
W20260928 14:45:13.226655 23713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:13.227464 23705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:13.228588 23705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:13.229831 23705 hybrid_clock.cc:648] HybridClock initialized: now 1790606713229776 us; error 28 us; skew 500 ppm
Sep 28 14:45:13 dist-test-slave-0n27 krb5kdc[23692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606713, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:13.232563 23705 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:45:13.233345 23705 webserver.cc:460] Webserver started at http://127.1.201.62:46519/ using document root <none> and password file <none>
I20260928 14:45:13.233914 23705 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:13.233980 23705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:13.234145 23705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:13.236121 23705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "6903755a5d354a4bb5d746579ad5d7cc"
format_stamp: "Formatted at 2026-09-28 14:45:13 on dist-test-slave-0n27"
server_key: "e60d9d11bfdce5c0bdea364e114548bc"
server_key_iv: "7b05ee244189501b54a563ce770714a5"
server_key_version: "encryptionkey@0"
I20260928 14:45:13.236600 23705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "6903755a5d354a4bb5d746579ad5d7cc"
format_stamp: "Formatted at 2026-09-28 14:45:13 on dist-test-slave-0n27"
server_key: "e60d9d11bfdce5c0bdea364e114548bc"
server_key_iv: "7b05ee244189501b54a563ce770714a5"
server_key_version: "encryptionkey@0"
I20260928 14:45:13.240053 23705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:45:13.242285 23720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:13.243412 23705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:45:13.243508 23705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "6903755a5d354a4bb5d746579ad5d7cc"
format_stamp: "Formatted at 2026-09-28 14:45:13 on dist-test-slave-0n27"
server_key: "e60d9d11bfdce5c0bdea364e114548bc"
server_key_iv: "7b05ee244189501b54a563ce770714a5"
server_key_version: "encryptionkey@0"
I20260928 14:45:13.243584 23705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:13.247669 23705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:13.250582 23705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:13.250725 23705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:13.256762 23705 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:36895
I20260928 14:45:13.256809 23772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:36895 every 8 connection(s)
I20260928 14:45:13.257583 23705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:13.260493 23773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:13.265548 23773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc: Bootstrap starting.
I20260928 14:45:13.266388 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 23705
I20260928 14:45:13.266517 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:13.266772 1828 external_mini_cluster.cc:1428] Setting key cc27b73b95f6cfea97c01c643b6f6296
I20260928 14:45:13.268013 23773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:13.268709 23773 log.cc:824] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:13.270790 23773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc: No bootstrap required, opened a new log
I20260928 14:45:13.272367 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:13.267741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44979 (local address 127.1.201.62:36895)
0928 14:45:13.268175 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:13.268202 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:13.268234 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:45:13.268866 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:13.268883 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:13.269222 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:13.269615 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:13.270691 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:13.270700 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:13.270715 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:13.270726 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:13.270803 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:13.270828 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:13.271268 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:13.271344 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 14:45:13.271498 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":122,"threads_started":1}
I20260928 14:45:13.274814 23773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6903755a5d354a4bb5d746579ad5d7cc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 36895 } }
I20260928 14:45:13.275045 23773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:13.275071 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:13.275125 23773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6903755a5d354a4bb5d746579ad5d7cc, State: Initialized, Role: FOLLOWER
I20260928 14:45:13.275545 23773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6903755a5d354a4bb5d746579ad5d7cc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 36895 } }
I20260928 14:45:13.275700 23773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:13.275784 23773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:13.275875 23773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:13.276816 23773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6903755a5d354a4bb5d746579ad5d7cc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 36895 } }
I20260928 14:45:13.277156 23773 leader_election.cc:302] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6903755a5d354a4bb5d746579ad5d7cc; no voters:
I20260928 14:45:13.277480 23773 leader_election.cc:288] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:13.277580 23783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:13.277850 23783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [term 1 LEADER]: Becoming Leader. State: Replica: 6903755a5d354a4bb5d746579ad5d7cc, State: Running, Role: LEADER
I20260928 14:45:13.278432 23783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6903755a5d354a4bb5d746579ad5d7cc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 36895 } }
I20260928 14:45:13.278685 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:13.277092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46155 (local address 127.1.201.62:36895)
0928 14:45:13.277205 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:13.277209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:13.277363 (+ 154us) server_negotiation.cc:407] Connection header received
0928 14:45:13.277521 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:13.277524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:13.277554 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:13.277632 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:13.278302 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:13.278305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:13.278307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:13.278318 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:13.278340 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:13.278351 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:13.278430 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:13.278550 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 14:45:13.278583 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:45:13.278687 23773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:13.280874 23784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6903755a5d354a4bb5d746579ad5d7cc. Latest consensus state: current_term: 1 leader_uuid: "6903755a5d354a4bb5d746579ad5d7cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6903755a5d354a4bb5d746579ad5d7cc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 36895 } } }
I20260928 14:45:13.280902 23785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6903755a5d354a4bb5d746579ad5d7cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6903755a5d354a4bb5d746579ad5d7cc" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 36895 } } }
I20260928 14:45:13.281210 23785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:13.281208 23784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc [sys.catalog]: This master's current role is: LEADER
W20260928 14:45:13.283201 23797 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:45:13.283280 23797 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:13.283342 23798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:13.283974 23798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:13.288884 23798 catalog_manager.cc:1347] Generated new cluster ID: 153d2b0e1fdd4f4c9c5ceb9306a37dfc
I20260928 14:45:13.288938 23798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:13.312028 23798 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:13.312445 23798 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:13.317344 23798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6903755a5d354a4bb5d746579ad5d7cc: Generated new TSK 0
I20260928 14:45:13.318056 23798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:13.321985 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 23705
2026-09-28T14:45:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:13 dist-test-slave-0n27 krb5kdc[23804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:13 dist-test-slave-0n27 krb5kdc[23804](info): set up 2 sockets
Sep 28 14:45:13 dist-test-slave-0n27 krb5kdc[23804](info): commencing operation
krb5kdc: starting...
W20260928 14:45:14.326759 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:14 dist-test-slave-0n27 krb5kdc[23804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:45:14.440850 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40061
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:40061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:14.546723 23817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:14.546957 23817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:14.546995 23817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:14.550782 23817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:14.550832 23817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:14.550851 23817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:14.550868 23817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:14.550881 23817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:14.556186 23817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:40061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:14.557119 23817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:14.558089 23817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:14.566289 23825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:14.566254 23823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:14.566264 23822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:14.566430 23817 server_base.cc:1034] running on GCE node
I20260928 14:45:14.567016 23817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:14.568076 23817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:14.569238 23817 hybrid_clock.cc:648] HybridClock initialized: now 1790606714569206 us; error 44 us; skew 500 ppm
Sep 28 14:45:14 dist-test-slave-0n27 krb5kdc[23804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606714, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:14.578531 23817 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:45:14.579486 23817 webserver.cc:460] Webserver started at http://127.1.201.62:33827/ using document root <none> and password file <none>
I20260928 14:45:14.580044 23817 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:14.580104 23817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:14.580350 23817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:14.582516 23817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "71911e11711d4a7f9c25447fcc9661e3"
format_stamp: "Formatted at 2026-09-28 14:45:14 on dist-test-slave-0n27"
server_key: "d8438dda40e7b276e724e94b8e1dd6c5"
server_key_iv: "7081020455284dd5ebcbd271d742aa0d"
server_key_version: "encryptionkey@0"
I20260928 14:45:14.583071 23817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "71911e11711d4a7f9c25447fcc9661e3"
format_stamp: "Formatted at 2026-09-28 14:45:14 on dist-test-slave-0n27"
server_key: "d8438dda40e7b276e724e94b8e1dd6c5"
server_key_iv: "7081020455284dd5ebcbd271d742aa0d"
server_key_version: "encryptionkey@0"
I20260928 14:45:14.586673 23817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:45:14.589010 23832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:14.590292 23817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:45:14.590428 23817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "71911e11711d4a7f9c25447fcc9661e3"
format_stamp: "Formatted at 2026-09-28 14:45:14 on dist-test-slave-0n27"
server_key: "d8438dda40e7b276e724e94b8e1dd6c5"
server_key_iv: "7081020455284dd5ebcbd271d742aa0d"
server_key_version: "encryptionkey@0"
I20260928 14:45:14.590538 23817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:14.606491 23817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:14.609045 23817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:14.609248 23817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:14.616276 23817 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:40061
I20260928 14:45:14.616302 23884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:40061 every 8 connection(s)
I20260928 14:45:14.617167 23817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:14.619694 23885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:14.624848 23885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3: Bootstrap starting.
I20260928 14:45:14.626585 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 23817
I20260928 14:45:14.626734 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:14.627000 1828 external_mini_cluster.cc:1428] Setting key f269a7f06acd985ccd0ec361a437fcef
I20260928 14:45:14.627429 23885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:14.628276 23885 log.cc:824] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:14.630122 23885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3: No bootstrap required, opened a new log
I20260928 14:45:14.633342 23885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71911e11711d4a7f9c25447fcc9661e3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40061 } }
I20260928 14:45:14.633594 23885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:14.633661 23885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71911e11711d4a7f9c25447fcc9661e3, State: Initialized, Role: FOLLOWER
I20260928 14:45:14.634172 23885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71911e11711d4a7f9c25447fcc9661e3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40061 } }
I20260928 14:45:14.634359 23885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:14.634436 23885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:14.634544 23885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 14:45:14 dist-test-slave-0n27 krb5kdc[23804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:45:14.635625 23885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71911e11711d4a7f9c25447fcc9661e3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40061 } }
I20260928 14:45:14.635968 23885 leader_election.cc:302] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71911e11711d4a7f9c25447fcc9661e3; no voters:
I20260928 14:45:14.636276 23885 leader_election.cc:288] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:14.636364 23890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:14.636575 23890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [term 1 LEADER]: Becoming Leader. State: Replica: 71911e11711d4a7f9c25447fcc9661e3, State: Running, Role: LEADER
I20260928 14:45:14.636897 23890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71911e11711d4a7f9c25447fcc9661e3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40061 } }
I20260928 14:45:14.637516 23885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:14.638559 23891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71911e11711d4a7f9c25447fcc9661e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71911e11711d4a7f9c25447fcc9661e3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40061 } } }
I20260928 14:45:14.638762 23891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:14.639220 23892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71911e11711d4a7f9c25447fcc9661e3. Latest consensus state: current_term: 1 leader_uuid: "71911e11711d4a7f9c25447fcc9661e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71911e11711d4a7f9c25447fcc9661e3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40061 } } }
I20260928 14:45:14.639401 23892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:14.639333 23899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:14.642334 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:14.628064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46469 (local address 127.1.201.62:40061)
0928 14:45:14.628505 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:14.628520 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:14.628553 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:45:14.629333 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:14.629350 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:14.629671 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:14.630054 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:14.630862 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:14.631829 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:14.632678 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:14.633132 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:14.633296 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:14.636277 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:14.636282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:14.636295 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:14.636322 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:14.638696 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:14.639273 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:14.639278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:14.639285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:14.639427 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:14.639756 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:14.639759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:14.639761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:14.640242 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:14.641106 (+ 864us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:14.641264 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 14:45:14.641572 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":124,"threads_started":1}
I20260928 14:45:14.642748 23899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:14.645299 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:14.648756 23899 catalog_manager.cc:1347] Generated new cluster ID: 5ba352f5e46344989fb28354d6d7080d
I20260928 14:45:14.648813 23899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:14.655776 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:14.647589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45915 (local address 127.1.201.62:40061)
0928 14:45:14.647702 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:14.647706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:14.647997 (+ 291us) server_negotiation.cc:407] Connection header received
0928 14:45:14.648112 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:14.648114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:14.648159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:14.648241 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:14.649033 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:14.649511 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:14.650620 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:14.650995 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:14.651112 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:14.652547 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:14.652551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:14.652554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:14.652578 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:14.654167 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:14.654598 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:14.654601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:14.654603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:14.654694 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:14.654959 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:14.654962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:14.654964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:14.655165 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:14.655293 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:14.655499 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 14:45:14.655621 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:45:14.674139 23899 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:14.675132 23899 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:14.689702 23899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71911e11711d4a7f9c25447fcc9661e3: Generated new TSK 0
I20260928 14:45:14.690536 23899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:14.703429 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 23817
2026-09-28T14:45:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:14 dist-test-slave-0n27 krb5kdc[23916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:14 dist-test-slave-0n27 krb5kdc[23916](info): set up 2 sockets
Sep 28 14:45:14 dist-test-slave-0n27 krb5kdc[23916](info): commencing operation
krb5kdc: starting...
W20260928 14:45:15.718999 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:15 dist-test-slave-0n27 krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:45:15.834715 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:41481
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:15.954918 23929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:15.955169 23929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:15.955235 23929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:15.959147 23929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:15.959223 23929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:15.959267 23929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:15.959298 23929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:15.959349 23929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:15.964507 23929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:15.965488 23929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:15.966497 23929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:15.974758 23937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:15.974804 23935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:15.974807 23929 server_base.cc:1034] running on GCE node
W20260928 14:45:15.974758 23934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:15.975507 23929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:15.976524 23929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:15.977708 23929 hybrid_clock.cc:648] HybridClock initialized: now 1790606715977688 us; error 32 us; skew 500 ppm
Sep 28 14:45:15 dist-test-slave-0n27 krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606715, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:15.980259 23929 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:45:15.981134 23929 webserver.cc:460] Webserver started at http://127.1.201.62:38439/ using document root <none> and password file <none>
I20260928 14:45:15.981673 23929 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:15.981762 23929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:15.981984 23929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:15.983928 23929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "952ca1ee0d6a44408c94623f80c7759e"
format_stamp: "Formatted at 2026-09-28 14:45:15 on dist-test-slave-0n27"
server_key: "65c4dad207375a61ecac11ce2d5afbc3"
server_key_iv: "a5f33e9d9e35ee9c7fa378d9633d6f66"
server_key_version: "encryptionkey@0"
I20260928 14:45:15.984478 23929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "952ca1ee0d6a44408c94623f80c7759e"
format_stamp: "Formatted at 2026-09-28 14:45:15 on dist-test-slave-0n27"
server_key: "65c4dad207375a61ecac11ce2d5afbc3"
server_key_iv: "a5f33e9d9e35ee9c7fa378d9633d6f66"
server_key_version: "encryptionkey@0"
I20260928 14:45:15.988144 23929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 14:45:15.990572 23944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:15.991741 23929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260928 14:45:15.991896 23929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "952ca1ee0d6a44408c94623f80c7759e"
format_stamp: "Formatted at 2026-09-28 14:45:15 on dist-test-slave-0n27"
server_key: "65c4dad207375a61ecac11ce2d5afbc3"
server_key_iv: "a5f33e9d9e35ee9c7fa378d9633d6f66"
server_key_version: "encryptionkey@0"
I20260928 14:45:15.992007 23929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:16.008991 23929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:16.011767 23929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:16.011986 23929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:16.018824 23929 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:41481
I20260928 14:45:16.018847 23996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:41481 every 8 connection(s)
I20260928 14:45:16.019695 23929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:16.020426 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 23929
I20260928 14:45:16.020565 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:16.020871 1828 external_mini_cluster.cc:1428] Setting key 4feef0f82d1d704bc6863be40770d1e9
I20260928 14:45:16.022723 23997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:16.029371 23997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e: Bootstrap starting.
I20260928 14:45:16.030478 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:16.022231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48009 (local address 127.1.201.62:41481)
0928 14:45:16.022666 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:16.022692 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:16.022734 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:45:16.023541 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:16.023561 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:16.023994 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:16.024354 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:16.024733 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:16.026225 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:16.027094 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:16.027774 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:16.027931 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:16.028657 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:16.028663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:16.028686 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:16.029134 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:16.029195 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:16.029224 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:16.029319 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:16.029410 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:45:16.029606 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":146,"threads_started":1}
I20260928 14:45:16.031880 23997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:16.032681 23997 log.cc:824] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:16.033275 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:16.034890 23997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e: No bootstrap required, opened a new log
I20260928 14:45:16.039204 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:16.035419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43443 (local address 127.1.201.62:41481)
0928 14:45:16.035537 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:16.035541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:16.035730 (+ 189us) server_negotiation.cc:407] Connection header received
0928 14:45:16.035840 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:16.035842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:16.035875 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:16.035934 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:16.036173 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:16.036624 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:16.037764 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:16.038024 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:16.038097 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:16.038823 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:16.038825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:16.038827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:16.038837 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:16.038859 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:16.038868 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:16.038923 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:16.039032 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 14:45:16.039072 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:45:16.039315 23997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ca1ee0d6a44408c94623f80c7759e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41481 } }
I20260928 14:45:16.039594 23997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:16.039660 23997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952ca1ee0d6a44408c94623f80c7759e, State: Initialized, Role: FOLLOWER
I20260928 14:45:16.040139 23997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ca1ee0d6a44408c94623f80c7759e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41481 } }
I20260928 14:45:16.040299 23997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:16.040365 23997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:16.040457 23997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:16.041288 23997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ca1ee0d6a44408c94623f80c7759e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41481 } }
I20260928 14:45:16.041615 23997 leader_election.cc:302] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 952ca1ee0d6a44408c94623f80c7759e; no voters:
I20260928 14:45:16.041934 23997 leader_election.cc:288] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:16.042042 24007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:16.042285 24007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [term 1 LEADER]: Becoming Leader. State: Replica: 952ca1ee0d6a44408c94623f80c7759e, State: Running, Role: LEADER
I20260928 14:45:16.042658 24007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ca1ee0d6a44408c94623f80c7759e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41481 } }
I20260928 14:45:16.043147 23997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:16.044312 24009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 952ca1ee0d6a44408c94623f80c7759e. Latest consensus state: current_term: 1 leader_uuid: "952ca1ee0d6a44408c94623f80c7759e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ca1ee0d6a44408c94623f80c7759e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41481 } } }
I20260928 14:45:16.044432 24009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:16.044834 24008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "952ca1ee0d6a44408c94623f80c7759e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ca1ee0d6a44408c94623f80c7759e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 41481 } } }
I20260928 14:45:16.045506 24008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e [sys.catalog]: This master's current role is: LEADER
W20260928 14:45:16.047675 24022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:45:16.047771 24022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:16.047847 24016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:16.048516 24016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:16.053918 24016 catalog_manager.cc:1347] Generated new cluster ID: a9df2df218df4322b7edc6a7e04ed337
I20260928 14:45:16.053980 24016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:16.070667 24016 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:16.071520 24016 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:16.079368 24016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 952ca1ee0d6a44408c94623f80c7759e: Generated new TSK 0
I20260928 14:45:16.080020 24016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:16.084757 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 23929
2026-09-28T14:45:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:16 dist-test-slave-0n27 krb5kdc[24028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:16 dist-test-slave-0n27 krb5kdc[24028](info): set up 2 sockets
Sep 28 14:45:16 dist-test-slave-0n27 krb5kdc[24028](info): commencing operation
krb5kdc: starting...
W20260928 14:45:17.094977 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:17 dist-test-slave-0n27 krb5kdc[24028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:45:17.212066 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:37787
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:37787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:17.318645 24041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:17.318923 24041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:17.319016 24041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:17.322932 24041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:17.323014 24041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:17.323051 24041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:17.323107 24041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:17.323143 24041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:17.328470 24041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:37787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:37787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:17.329466 24041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:17.330456 24041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:17.338511 24049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:17.338627 24041 server_base.cc:1034] running on GCE node
W20260928 14:45:17.338496 24046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:17.338795 24047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:17.339345 24041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:17.340310 24041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:17.341475 24041 hybrid_clock.cc:648] HybridClock initialized: now 1790606717341458 us; error 29 us; skew 500 ppm
Sep 28 14:45:17 dist-test-slave-0n27 krb5kdc[24028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606717, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:17.344023 24041 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:45:17.344914 24041 webserver.cc:460] Webserver started at http://127.1.201.62:42681/ using document root <none> and password file <none>
I20260928 14:45:17.345445 24041 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:17.345501 24041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:17.345724 24041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:17.347731 24041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "3ce6b0c909e143de80220674d1b3e25e"
format_stamp: "Formatted at 2026-09-28 14:45:17 on dist-test-slave-0n27"
server_key: "eafbd0b336eeaacb5795db677d7da8a1"
server_key_iv: "4bec71789f366ba6ede654420081a36c"
server_key_version: "encryptionkey@0"
I20260928 14:45:17.348287 24041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "3ce6b0c909e143de80220674d1b3e25e"
format_stamp: "Formatted at 2026-09-28 14:45:17 on dist-test-slave-0n27"
server_key: "eafbd0b336eeaacb5795db677d7da8a1"
server_key_iv: "4bec71789f366ba6ede654420081a36c"
server_key_version: "encryptionkey@0"
I20260928 14:45:17.351853 24041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:45:17.354177 24056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:17.355336 24041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:45:17.355482 24041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "3ce6b0c909e143de80220674d1b3e25e"
format_stamp: "Formatted at 2026-09-28 14:45:17 on dist-test-slave-0n27"
server_key: "eafbd0b336eeaacb5795db677d7da8a1"
server_key_iv: "4bec71789f366ba6ede654420081a36c"
server_key_version: "encryptionkey@0"
I20260928 14:45:17.355609 24041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:17.359922 24041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:17.362668 24041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:17.362839 24041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:17.369213 24041 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:37787
I20260928 14:45:17.369230 24108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:37787 every 8 connection(s)
I20260928 14:45:17.370062 24041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:17.372561 24109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:17.377554 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 24041
I20260928 14:45:17.377696 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:17.377885 24109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e: Bootstrap starting.
I20260928 14:45:17.378031 1828 external_mini_cluster.cc:1428] Setting key c0d1fa991cc480e17dbff14d5757828b
I20260928 14:45:17.380276 24109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:17.380976 24109 log.cc:824] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:17.383168 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:17.379327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35105 (local address 127.1.201.62:37787)
0928 14:45:17.379736 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:17.379751 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:17.379785 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:45:17.380417 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:17.380440 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:17.380723 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:17.381041 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:17.381856 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:17.381865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:17.381894 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:17.381905 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:17.381970 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:17.381994 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:17.382130 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:17.382219 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 14:45:17.382354 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":127,"threads_started":1}
I20260928 14:45:17.383200 24109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e: No bootstrap required, opened a new log
I20260928 14:45:17.385725 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:17.386749 24109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b0c909e143de80220674d1b3e25e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37787 } }
I20260928 14:45:17.387048 24109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:17.387125 24109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce6b0c909e143de80220674d1b3e25e, State: Initialized, Role: FOLLOWER
I20260928 14:45:17.387570 24109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b0c909e143de80220674d1b3e25e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37787 } }
I20260928 14:45:17.387807 24109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:17.387888 24109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:17.387984 24109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:17.389035 24109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b0c909e143de80220674d1b3e25e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37787 } }
I20260928 14:45:17.389240 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:17.387662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41515 (local address 127.1.201.62:37787)
0928 14:45:17.387770 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:17.387774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:17.387941 (+ 167us) server_negotiation.cc:407] Connection header received
0928 14:45:17.388070 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:17.388075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:17.388109 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:17.388175 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:17.388881 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:17.388884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:17.388886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:17.388897 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:17.388922 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:17.388934 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:17.389023 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:17.389095 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 14:45:17.389133 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:45:17.389411 24109 leader_election.cc:302] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ce6b0c909e143de80220674d1b3e25e; no voters:
I20260928 14:45:17.389760 24109 leader_election.cc:288] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:17.389873 24119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:17.390030 24119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [term 1 LEADER]: Becoming Leader. State: Replica: 3ce6b0c909e143de80220674d1b3e25e, State: Running, Role: LEADER
I20260928 14:45:17.390342 24119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b0c909e143de80220674d1b3e25e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37787 } }
I20260928 14:45:17.390991 24109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:17.392033 24120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ce6b0c909e143de80220674d1b3e25e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b0c909e143de80220674d1b3e25e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37787 } } }
I20260928 14:45:17.392076 24121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ce6b0c909e143de80220674d1b3e25e. Latest consensus state: current_term: 1 leader_uuid: "3ce6b0c909e143de80220674d1b3e25e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b0c909e143de80220674d1b3e25e" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 37787 } } }
I20260928 14:45:17.392223 24120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:17.392230 24121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:17.392619 24128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:17.395565 24128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:17.400801 24128 catalog_manager.cc:1347] Generated new cluster ID: 23b7f2189a8a44eba68a2b4d6e1b9b53
I20260928 14:45:17.400895 24128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:17.411610 24128 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:17.411912 24128 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:17.420686 24128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ce6b0c909e143de80220674d1b3e25e: Generated new TSK 0
I20260928 14:45:17.421484 24128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:17.435302 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 24041
2026-09-28T14:45:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1350 ms)
[----------] 18 tests from AuthTokenIssuingTest (24592 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:17 dist-test-slave-0n27 krb5kdc[24140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:17 dist-test-slave-0n27 krb5kdc[24140](info): set up 2 sockets
Sep 28 14:45:17 dist-test-slave-0n27 krb5kdc[24140](info): commencing operation
krb5kdc: starting...
W20260928 14:45:18.434859 1828 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:18 dist-test-slave-0n27 krb5kdc[24140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:45:18.550019 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40391
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:40391,127.1.201.61:44861,127.1.201.60:41557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:18.657371 24153 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:45:18.657624 24153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:18.657687 24153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:18.657759 24153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:18.661628 24153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:18.661684 24153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:18.661705 24153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:18.661720 24153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:18.661800 24153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:18.666782 24153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:40391,127.1.201.61:44861,127.1.201.60:41557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:40391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:18.667752 24153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:18.668711 24153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:18.677125 24161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:18.677173 24153 server_base.cc:1034] running on GCE node
W20260928 14:45:18.677093 24158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:18.677357 24159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:18.677946 24153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:18.679018 24153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:18.680174 24153 hybrid_clock.cc:648] HybridClock initialized: now 1790606718680162 us; error 25 us; skew 500 ppm
Sep 28 14:45:18 dist-test-slave-0n27 krb5kdc[24140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:18.682893 24153 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:45:18.683714 24153 webserver.cc:460] Webserver started at http://127.1.201.62:41525/ using document root <none> and password file <none>
I20260928 14:45:18.684257 24153 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:18.684342 24153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:18.684571 24153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:18.686697 24153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "7a1bd4ade6a649ccb4080f548cd3eab3"
format_stamp: "Formatted at 2026-09-28 14:45:18 on dist-test-slave-0n27"
server_key: "c6162439b97791e6b1c896054bfa7dac"
server_key_iv: "e33343ad13fb0edb5e27f988771e0ac9"
server_key_version: "encryptionkey@0"
I20260928 14:45:18.687254 24153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "7a1bd4ade6a649ccb4080f548cd3eab3"
format_stamp: "Formatted at 2026-09-28 14:45:18 on dist-test-slave-0n27"
server_key: "c6162439b97791e6b1c896054bfa7dac"
server_key_iv: "e33343ad13fb0edb5e27f988771e0ac9"
server_key_version: "encryptionkey@0"
I20260928 14:45:18.690898 24153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:45:18.693316 24168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:18.694630 24153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 14:45:18.694772 24153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "7a1bd4ade6a649ccb4080f548cd3eab3"
format_stamp: "Formatted at 2026-09-28 14:45:18 on dist-test-slave-0n27"
server_key: "c6162439b97791e6b1c896054bfa7dac"
server_key_iv: "e33343ad13fb0edb5e27f988771e0ac9"
server_key_version: "encryptionkey@0"
I20260928 14:45:18.694880 24153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:18.715514 24153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:18.718561 24153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:18.718776 24153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:18.725718 24153 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:40391
I20260928 14:45:18.726058 24221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:40391 every 8 connection(s)
I20260928 14:45:18.726698 24153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:18.729333 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:18.732903 24222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:18.735910 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 24153
I20260928 14:45:18.736052 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:18.736354 1828 external_mini_cluster.cc:1428] Setting key ec3c0e13935dbbcc9be2bc2f61d05786
Sep 28 14:45:18 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.61@KRBTEST.COM" created.
I20260928 14:45:18.748311 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.735037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40391 (local address 127.0.0.1:54778)
0928 14:45:18.735651 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.735716 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:18.736683 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:18.737356 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:18.737361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:18.737823 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:18.738074 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:18.738087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.739207 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.739211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:18.739931 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:18.739940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.740579 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.740583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:18.740800 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:18.741468 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:18.741484 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:18.743192 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:18.745219 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:18.745228 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:18.745237 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:18.745456 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:18.745824 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:18.745827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:18.745829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:18.745968 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:18.746631 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:18.746636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:18.746778 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 14:45:18.746936 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 14:45:18.747325 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":200,"spinlock_wait_cycles":53504,"thread_start_us":154,"threads_started":1}
I20260928 14:45:18.748311 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.735037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.1.201.62:40391)
0928 14:45:18.735560 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:18.735577 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:18.735821 (+ 244us) server_negotiation.cc:407] Connection header received
0928 14:45:18.736997 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:18.737001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:18.737176 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:18.737360 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:18.738227 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.739083 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.740053 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.740423 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.740717 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:18.743349 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:18.743355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:18.743363 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:18.743383 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:18.745089 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:18.745568 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:18.745573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:18.745575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:18.745677 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:18.746063 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:18.746068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:18.746070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:18.746520 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:18.746651 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:18.747008 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 14:45:18.747276 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":353,"spinlock_wait_cycles":768,"thread_start_us":135,"threads_started":1}
I20260928 14:45:18.750833 24222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:18.751586 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.751316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:44861 (local address 127.0.0.1:45556)
0928 14:45:18.751454 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.751529 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 14:45:18.751780 24173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.201.61:44861: connect: Connection refused (error 111)
I20260928 14:45:18.752683 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.752452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:44861 (local address 127.0.0.1:45564)
0928 14:45:18.752574 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.752622 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 14:45:18.752848 24222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.61:44861: Network error: Client connection negotiation failed: client connection to 127.1.201.61:44861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab.
Entry for principal kudu/127.1.201.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab.
I20260928 14:45:18.762210 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.61:44861
--webserver_interface=127.1.201.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab
--principal=kudu/127.1.201.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:40391,127.1.201.61:44861,127.1.201.60:41557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:45:18.802062 24222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } attempt: 1
I20260928 14:45:18.803009 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.802587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:44861 (local address 127.0.0.1:45578)
0928 14:45:18.802797 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.802891 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 14:45:18.803826 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.803583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:44861 (local address 127.0.0.1:45586)
0928 14:45:18.803708 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.803763 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 14:45:18.803985 24222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.61:44861: Network error: Client connection negotiation failed: client connection to 127.1.201.61:44861: connect: Connection refused (error 111)
W20260928 14:45:18.870386 24229 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:45:18.870644 24229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:18.870666 24229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:18.870776 24229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:45:18.872167 24222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } attempt: 2
I20260928 14:45:18.873015 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.872697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:44861 (local address 127.0.0.1:45600)
0928 14:45:18.872859 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.872929 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 14:45:18.873799 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.873567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:44861 (local address 127.0.0.1:45604)
0928 14:45:18.873672 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.873723 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:45:18.873937 24222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.61:44861: Network error: Client connection negotiation failed: client connection to 127.1.201.61:44861: connect: Connection refused (error 111)
W20260928 14:45:18.875164 24229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:18.875222 24229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:18.875240 24229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:18.875255 24229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:18.875268 24229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:18.880811 24229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:40391,127.1.201.61:44861,127.1.201.60:41557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab
--principal=kudu/127.1.201.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.61:44861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.201.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:18.881752 24229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:18.882757 24229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:18.890946 24234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:18.891008 24229 server_base.cc:1034] running on GCE node
W20260928 14:45:18.890933 24235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:18.890951 24237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:18.891731 24229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:18.892673 24229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:18.893862 24229 hybrid_clock.cc:648] HybridClock initialized: now 1790606718893808 us; error 42 us; skew 500 ppm
Sep 28 14:45:18 dist-test-slave-0n27 krb5kdc[24140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:18.896515 24229 init.cc:377] Logged in from keytab as kudu/127.1.201.61@KRBTEST.COM (short username kudu)
I20260928 14:45:18.897416 24229 webserver.cc:460] Webserver started at http://127.1.201.61:33433/ using document root <none> and password file <none>
I20260928 14:45:18.898046 24229 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:18.898150 24229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:18.898381 24229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:18.900389 24229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/data/instance:
uuid: "26be03f4762c42b6b47d10fa8a245bec"
format_stamp: "Formatted at 2026-09-28 14:45:18 on dist-test-slave-0n27"
server_key: "6549da55d1d52d1530495cdf20a12f59"
server_key_iv: "6b120d20e4a2f7ff54fc15cc5502ae3e"
server_key_version: "encryptionkey@0"
I20260928 14:45:18.900975 24229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/wal/instance:
uuid: "26be03f4762c42b6b47d10fa8a245bec"
format_stamp: "Formatted at 2026-09-28 14:45:18 on dist-test-slave-0n27"
server_key: "6549da55d1d52d1530495cdf20a12f59"
server_key_iv: "6b120d20e4a2f7ff54fc15cc5502ae3e"
server_key_version: "encryptionkey@0"
I20260928 14:45:18.904455 24229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 14:45:18.907130 24244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:18.908308 24229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:45:18.908450 24229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/wal
uuid: "26be03f4762c42b6b47d10fa8a245bec"
format_stamp: "Formatted at 2026-09-28 14:45:18 on dist-test-slave-0n27"
server_key: "6549da55d1d52d1530495cdf20a12f59"
server_key_iv: "6b120d20e4a2f7ff54fc15cc5502ae3e"
server_key_version: "encryptionkey@0"
I20260928 14:45:18.908557 24229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:18.945705 24229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:18.948578 24229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:18.948778 24229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:18.955462 24229 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.61:44861
I20260928 14:45:18.955760 24297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.61:44861 every 8 connection(s)
I20260928 14:45:18.956475 24229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
I20260928 14:45:18.958020 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 24229
I20260928 14:45:18.958173 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-1/wal/instance
I20260928 14:45:18.958498 1828 external_mini_cluster.cc:1428] Setting key 4f63f07ffbff073f1a6376f50a8b0573
I20260928 14:45:18.959290 24298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:18.963465 24298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:18.965095 24222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } attempt: 3
WARNING: no policy specified for kudu/127.1.201.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.60@KRBTEST.COM" created.
Sep 28 14:45:18 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
Sep 28 14:45:18 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:45:18.977599 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.965519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:44861 (local address 127.0.0.1:45608)
0928 14:45:18.965675 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.965707 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:18.965821 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:18.967511 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:18.967514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:18.967782 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:18.967887 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:18.967894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.969396 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.969400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:18.970203 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:18.970214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.971076 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.971080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:18.971182 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:18.971720 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:18.971740 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:18.973353 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:18.975757 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:18.975761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:18.975763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:18.976016 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:18.976402 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:18.976405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:18.976407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:18.976496 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:18.977058 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:18.977063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:18.977162 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 14:45:18.977284 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 14:45:18.977416 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"spinlock_wait_cycles":1920}
I20260928 14:45:18.978200 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.965326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.1.201.62:40391)
0928 14:45:18.965475 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:18.965480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:18.966514 (+ 1034us) server_negotiation.cc:407] Connection header received
0928 14:45:18.967301 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:18.967304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:18.967349 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:18.967422 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:18.968434 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.968971 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.970369 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.970617 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.970730 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:18.974044 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:18.974049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:18.974056 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:18.974099 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:18.975654 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:18.976096 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:18.976099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:18.976101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:18.976188 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:18.976546 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:18.976549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:18.976551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:18.976793 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:18.976897 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:18.977961 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0928 14:45:18.978071 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:45:18.978510 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.965807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40391 (local address 127.0.0.1:54786)
0928 14:45:18.966396 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.966438 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:18.967059 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:18.967495 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:18.967529 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:18.967964 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:18.968306 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:18.968319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.969082 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.969088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:18.970268 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:18.970280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.970936 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.970941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:18.971129 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:18.971922 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:18.971938 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:18.973916 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:18.975776 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:18.975784 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:18.975791 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:18.975988 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:18.976276 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:18.976281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:18.976283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:18.976426 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:18.976907 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:18.976912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:18.977027 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 14:45:18.977168 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 14:45:18.977498 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":36,"spinlock_wait_cycles":29056,"thread_start_us":169,"threads_started":1}
I20260928 14:45:18.978509 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.965790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.1.201.61:44861)
0928 14:45:18.966215 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:18.966235 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:18.966273 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:45:18.967059 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:18.967080 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:18.967330 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:18.967514 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:18.968017 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.969272 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.970341 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.970949 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.971124 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:18.973493 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:18.973500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:18.973512 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:18.973537 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:18.975614 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:18.976159 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:18.976165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:18.976168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:18.976302 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:18.976570 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:18.976575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:18.976577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:18.976955 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:18.977069 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:18.977871 (+ 802us) server_negotiation.cc:299] Negotiation successful
0928 14:45:18.978015 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":284,"spinlock_wait_cycles":2944,"thread_start_us":135,"threads_started":1}
I20260928 14:45:18.979699 24298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:18.980820 24222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:18.981626 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.981346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47110)
0928 14:45:18.981477 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.981561 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:45:18.982327 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.982062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47126)
0928 14:45:18.982242 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.982280 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260928 14:45:18.982506 24222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:41557: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab.
Entry for principal kudu/127.1.201.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab.
I20260928 14:45:18.984889 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.60:41557
--webserver_interface=127.1.201.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab
--principal=kudu/127.1.201.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:40391,127.1.201.61:44861,127.1.201.60:41557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:45:18 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
I20260928 14:45:18.989030 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.980163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:44861 (local address 127.0.0.1:45614)
0928 14:45:18.980509 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.980526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:18.980604 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:18.980832 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:18.980837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:18.981043 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:18.981151 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:18.981161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.982127 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.982132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:18.982835 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:18.982844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.983404 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.983407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:18.983523 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:18.984410 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:18.984420 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:18.985698 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:18.987624 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:18.987630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:18.987633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:18.987897 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:18.988170 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:18.988172 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:18.988173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:18.988248 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:18.988669 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:18.988671 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:18.988743 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 14:45:18.988820 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:45:18.988912 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
I20260928 14:45:18.989306 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.980392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.1.201.61:44861)
0928 14:45:18.980505 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:18.980510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:18.980594 (+ 84us) server_negotiation.cc:407] Connection header received
0928 14:45:18.980675 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:18.980678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:18.980720 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:18.980792 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:18.981280 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.982005 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.982979 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:18.983291 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:18.983380 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:18.985846 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:18.985853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:18.985859 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:18.985888 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:18.987465 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:18.987987 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:18.987993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:18.987998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:18.988088 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:18.988329 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:18.988335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:18.988340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:18.988572 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:18.988681 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:18.988981 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 14:45:18.989110 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:45:18.989955 24298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:18.990578 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.990358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47140)
0928 14:45:18.990461 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.990531 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:45:18.990789 24245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
I20260928 14:45:18.991583 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:18.991393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47144)
0928 14:45:18.991490 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:18.991535 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 14:45:18.991739 24298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:41557: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
I20260928 14:45:19.013691 24222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } attempt: 1
I20260928 14:45:19.014555 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.014203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47158)
0928 14:45:19.014378 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.014464 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:45:19.015342 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.015081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47172)
0928 14:45:19.015213 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.015268 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:45:19.015479 24222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:41557: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
I20260928 14:45:19.040979 24298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } attempt: 1
I20260928 14:45:19.041956 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.041584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47182)
0928 14:45:19.041767 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.041865 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 14:45:19.042694 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.042465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47196)
0928 14:45:19.042578 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.042631 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 14:45:19.042842 24298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:41557: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
I20260928 14:45:19.090634 24222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } attempt: 2
I20260928 14:45:19.091630 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.091257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47202)
0928 14:45:19.091475 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.091542 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 14:45:19.092376 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.092166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47216)
0928 14:45:19.092284 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.092329 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:45:19.092527 24222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:41557: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
W20260928 14:45:19.094239 24305 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:45:19.094520 24305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:19.094564 24305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:19.094642 24305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:19.099306 24305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:19.099360 24305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:19.099380 24305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:19.099431 24305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:19.099462 24305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:19.104693 24305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:40391,127.1.201.61:44861,127.1.201.60:41557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab
--principal=kudu/127.1.201.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.60:41557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.201.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:19.105675 24305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:19.106683 24305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:45:19.111014 24298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } attempt: 2
I20260928 14:45:19.111796 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.111500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47224)
0928 14:45:19.111667 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.111729 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:45:19.112532 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.112331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47232)
0928 14:45:19.112432 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.112476 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 14:45:19.112694 24298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:41557: Network error: Client connection negotiation failed: client connection to 127.1.201.60:41557: connect: Connection refused (error 111)
W20260928 14:45:19.115200 24311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:19.115226 24314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:19.115269 24305 server_base.cc:1034] running on GCE node
W20260928 14:45:19.115231 24312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:19.115978 24305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:19.116986 24305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:19.118166 24305 hybrid_clock.cc:648] HybridClock initialized: now 1790606719118136 us; error 41 us; skew 500 ppm
Sep 28 14:45:19 dist-test-slave-0n27 krb5kdc[24140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606719, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:19.120599 24305 init.cc:377] Logged in from keytab as kudu/127.1.201.60@KRBTEST.COM (short username kudu)
I20260928 14:45:19.121456 24305 webserver.cc:460] Webserver started at http://127.1.201.60:40431/ using document root <none> and password file <none>
I20260928 14:45:19.122044 24305 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:19.122164 24305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:19.122373 24305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:19.124353 24305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/data/instance:
uuid: "a91bebfb956540369d44a7ea349e7e79"
format_stamp: "Formatted at 2026-09-28 14:45:19 on dist-test-slave-0n27"
server_key: "c08fb20074c8096602c9833f7847fe51"
server_key_iv: "a7200e224664f0721fd64cafd46a1f63"
server_key_version: "encryptionkey@0"
I20260928 14:45:19.124902 24305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/wal/instance:
uuid: "a91bebfb956540369d44a7ea349e7e79"
format_stamp: "Formatted at 2026-09-28 14:45:19 on dist-test-slave-0n27"
server_key: "c08fb20074c8096602c9833f7847fe51"
server_key_iv: "a7200e224664f0721fd64cafd46a1f63"
server_key_version: "encryptionkey@0"
I20260928 14:45:19.128332 24305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 14:45:19.130613 24321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:19.131742 24305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:45:19.131883 24305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/wal
uuid: "a91bebfb956540369d44a7ea349e7e79"
format_stamp: "Formatted at 2026-09-28 14:45:19 on dist-test-slave-0n27"
server_key: "c08fb20074c8096602c9833f7847fe51"
server_key_iv: "a7200e224664f0721fd64cafd46a1f63"
server_key_version: "encryptionkey@0"
I20260928 14:45:19.131989 24305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:19.151654 24305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:19.154431 24305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:19.154668 24305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:19.161402 24305 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.60:41557
I20260928 14:45:19.161686 24373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.60:41557 every 8 connection(s)
I20260928 14:45:19.162379 24305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
I20260928 14:45:19.164973 24374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:19.168458 24374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:19.171296 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 24305
I20260928 14:45:19.171447 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606444849692-1828-0/minicluster-data/master-2/wal/instance
I20260928 14:45:19.171726 1828 external_mini_cluster.cc:1428] Setting key eaa5982a5ee2234c28e3a915526dd47b
Sep 28 14:45:19 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606719, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
Sep 28 14:45:19 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
I20260928 14:45:19.182134 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.170181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.1.201.62:40391)
0928 14:45:19.170314 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.170318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.171054 (+ 736us) server_negotiation.cc:407] Connection header received
0928 14:45:19.172067 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.172071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.172135 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.172237 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.173617 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.174281 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.175329 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.175549 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.175640 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.178182 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.178185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.178186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.178208 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.179898 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.180396 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.180403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.180406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.180495 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.180901 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.180904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.180905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.181177 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.181261 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.181772 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.181923 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:45:19.182966 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.172325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.1.201.60:41557)
0928 14:45:19.172720 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.172727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.172766 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:45:19.172905 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.172909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.173054 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.173185 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.174070 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.174968 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.175974 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.176289 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.176401 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.178868 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.178877 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.178889 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.178916 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.180525 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.181007 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.181012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.181014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.181093 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.181392 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.181395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.181396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.181609 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.181848 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.182067 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.182197 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"spinlock_wait_cycles":35200,"thread_start_us":133,"threads_started":1}
I20260928 14:45:19.182965 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.170463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:40391 (local address 127.0.0.1:54798)
0928 14:45:19.170892 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.170934 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:19.171846 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:19.172397 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:19.172404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:19.172872 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:19.173463 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:19.173485 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.174392 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.174394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:19.175217 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:19.175227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.175692 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.175696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:19.175886 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.176488 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:19.176504 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:19.178032 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:19.180005 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:19.180013 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:19.180020 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:19.180302 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:19.180611 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:19.180614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:19.180616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:19.180786 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:19.181289 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:19.181296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:19.181547 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 14:45:19.181725 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 14:45:19.182078 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260928 14:45:19.184249 24374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:45:19 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606719, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
Sep 28 14:45:19 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:45:19.191685 24222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } attempt: 3
I20260928 14:45:19.192790 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.184686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:44861 (local address 127.0.0.1:45620)
0928 14:45:19.184796 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.184809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:19.184876 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:19.185155 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:19.185157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:19.185316 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:19.185427 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:19.185433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.186229 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.186233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:19.186953 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:19.186965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.187562 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.187566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:19.187659 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.188088 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:19.188101 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:19.189398 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:19.191090 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:19.191092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:19.191093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:19.191360 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:19.191677 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:19.191680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:19.191682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:19.191772 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:19.192320 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:19.192323 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:19.192425 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 14:45:19.192529 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:45:19.192647 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:45:19.192835 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.184821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.1.201.61:44861)
0928 14:45:19.184955 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.184960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.184972 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:45:19.185011 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.185013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.185058 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.185122 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.185529 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.186137 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.187114 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.187465 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.187537 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.189528 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.189531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.189533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.189556 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.190980 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.191458 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.191465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.191469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.191562 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.191890 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.191893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.191895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.192140 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.192244 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.192581 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.192704 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:45:19.193676 24374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:19.195209 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.185457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.1.201.62:40391)
0928 14:45:19.185902 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.185904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.185913 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:45:19.185961 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.185963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.186007 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.186091 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.186703 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.187330 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.188294 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.188491 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.188601 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.190906 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.190911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.190915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.190942 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.193247 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.193877 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.193880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.193881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.194012 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.194269 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.194273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.194275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.194469 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.194664 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.194956 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.195061 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380}
Sep 28 14:45:19 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
Sep 28 14:45:19 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606719, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
I20260928 14:45:19.201423 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.192065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47246)
0928 14:45:19.192251 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.192265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:19.192329 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:19.192596 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:19.192601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:19.192777 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:19.192884 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:19.192890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.193737 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.193740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:19.194555 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:19.194565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.195104 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.195107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:19.195180 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.195608 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:19.195625 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:19.197050 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:19.199752 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:19.199757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:19.199761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:19.200003 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:19.200329 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:19.200334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:19.200337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:19.200440 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:19.200908 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:19.200912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:19.201007 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 14:45:19.201118 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:45:19.201253 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 14:45:19.201907 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.192142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.1.201.60:41557)
0928 14:45:19.192279 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.192284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.192298 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:45:19.192416 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.192422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.192468 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.192665 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.192980 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.193633 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.194664 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.194989 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.195363 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.197147 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.197151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.197153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.197173 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.199644 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.200134 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.200138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.200140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.200224 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.200530 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.200533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.200535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.200763 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.200875 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.201609 (+ 734us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.201730 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:45:19.202901 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.194196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47252)
0928 14:45:19.194727 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.194738 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:19.194798 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:19.195074 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:19.195076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:19.195227 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:19.195306 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:19.195314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.196277 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.196282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:19.196975 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:19.196983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.197645 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.197650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:19.197734 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.198174 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:19.198187 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:19.199412 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:19.201370 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:19.201373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:19.201375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:19.201605 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:19.201938 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:19.201943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:19.201946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:19.202034 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:19.202460 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:19.202463 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:19.202538 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 14:45:19.202660 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 14:45:19.202782 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474}
Sep 28 14:45:19 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
I20260928 14:45:19.203339 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.194297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.1.201.60:41557)
0928 14:45:19.194605 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.194615 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.194794 (+ 179us) server_negotiation.cc:407] Connection header received
0928 14:45:19.194861 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.194866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.194913 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.194980 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.195413 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.196160 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.197087 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.197431 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.197534 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.199536 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.199539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.199541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.199562 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.201271 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.201705 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.201709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.201711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.201843 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.202146 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.202150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.202153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.202363 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.202469 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.203088 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.203185 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260928 14:45:19.203845 24298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } attempt: 3
I20260928 14:45:19.205066 24222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3: Bootstrap starting.
I20260928 14:45:19.208117 24222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:19.208639 24374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79: Bootstrap starting.
Sep 28 14:45:19 dist-test-slave-0n27 krb5kdc[24140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
I20260928 14:45:19.209035 24222 log.cc:824] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:19.210547 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.196512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.1.201.61:44861)
0928 14:45:19.196611 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.196615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.196947 (+ 332us) server_negotiation.cc:407] Connection header received
0928 14:45:19.197042 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.197047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.197094 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.197159 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.198195 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.199238 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.200313 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.200634 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.200697 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.206589 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.206592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.206595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.206616 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.208503 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.208986 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.208990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.208992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.209070 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.209383 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.209386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.209388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.209606 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.209855 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.210263 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.210377 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:45:19.211234 24222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3: No bootstrap required, opened a new log
I20260928 14:45:19.211614 24374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:19.212718 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.204172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:41557 (local address 127.0.0.1:47258)
0928 14:45:19.204344 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:19.204359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:19.204439 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:19.204650 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:19.204653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:19.204806 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:19.204876 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:19.204881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.205595 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.205600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:19.206413 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:19.206424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.207064 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.207067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:19.207160 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.207634 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:19.207648 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:19.209133 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:19.211127 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:19.211132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:19.211134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:19.211384 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:19.211706 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:19.211711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:19.211713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:19.211797 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:19.212213 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:19.212217 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:19.212305 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 14:45:19.212412 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:45:19.212539 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 14:45:19.213099 24374 log.cc:824] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:19.213266 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.204255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.1.201.60:41557)
0928 14:45:19.204344 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.204349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.204401 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:45:19.204507 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.204510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.204551 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.204630 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.204978 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.205496 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.206548 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.206908 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.207061 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.209265 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.209269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.209271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.209293 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.210998 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.211527 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.211533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.211538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.211624 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.211887 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.211892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.211894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.212081 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.212211 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.213029 (+ 818us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.213137 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:45:19.214536 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:19.215274 24374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79: No bootstrap required, opened a new log
I20260928 14:45:19.215374 24222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } }
I20260928 14:45:19.215582 24222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1bd4ade6a649ccb4080f548cd3eab3, State: Initialized, Role: FOLLOWER
I20260928 14:45:19.215814 24196 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54814
I20260928 14:45:19.215967 24222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } }
I20260928 14:45:19.216207 24298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec: Bootstrap starting.
I20260928 14:45:19.217267 24222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:19.217335 24196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 14:45:19.217448 24196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:19.218582 24298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:19.218612 24196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } }
I20260928 14:45:19.219372 24298 log.cc:824] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:19.219252 24374 raft_consensus.cc:348] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } }
I20260928 14:45:19.219487 24196 leader_election.cc:288] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [CANDIDATE]: Term 1 election: Requested vote from peers 26be03f4762c42b6b47d10fa8a245bec (127.1.201.61:44861), a91bebfb956540369d44a7ea349e7e79 (127.1.201.60:41557)
I20260928 14:45:19.219494 24374 raft_consensus.cc:729] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91bebfb956540369d44a7ea349e7e79, State: Initialized, Role: FOLLOWER
I20260928 14:45:19.219795 24374 consensus_queue.cc:260] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } }
W20260928 14:45:19.220320 24175 tablet.cc:2369] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:45:19.220954 24374 sys_catalog.cc:564] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:19.221227 24395 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } } }
I20260928 14:45:19.221349 24395 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:19.221289 24348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a91bebfb956540369d44a7ea349e7e79"
I20260928 14:45:19.221490 24348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [term 0 FOLLOWER]: Advancing to term 1
W20260928 14:45:19.223044 24394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 14:45:19.223042 24382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } } }
W20260928 14:45:19.223160 24394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:19.223176 24382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:19.223327 24348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a1bd4ade6a649ccb4080f548cd3eab3 in term 1.
I20260928 14:45:19.223719 24169 leader_election.cc:302] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a1bd4ade6a649ccb4080f548cd3eab3, a91bebfb956540369d44a7ea349e7e79; no voters:
I20260928 14:45:19.223958 24382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:19.224136 24382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [term 1 LEADER]: Becoming Leader. State: Replica: 7a1bd4ade6a649ccb4080f548cd3eab3, State: Running, Role: LEADER
I20260928 14:45:19.224414 24382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } }
I20260928 14:45:19.225968 24382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a1bd4ade6a649ccb4080f548cd3eab3. Latest consensus state: current_term: 1 leader_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } } }
W20260928 14:45:19.226038 24407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:45:19.226097 24407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:19.226126 24382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:19.225342 24153 master_runner.cc:186] Error getting master registration for 127.1.201.61:44861: OK, instance_id { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" instance_seqno: 1790606718953548 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:45:19.228303 24153 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:45:19.229736 24305 master_runner.cc:186] Error getting master registration for 127.1.201.61:44861: OK, instance_id { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" instance_seqno: 1790606718953548 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:45:19.233043 24305 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:45:19.240135 24415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:19.240150 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.228867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.1.201.61:44861)
0928 14:45:19.228972 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.228976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.229932 (+ 956us) server_negotiation.cc:407] Connection header received
0928 14:45:19.230176 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.230180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.230225 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.230274 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.231699 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.232432 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.233822 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.234144 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.234246 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.236136 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.236140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.236143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.236169 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.238057 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.238897 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.238901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.238902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.238985 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.239275 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.239278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.239280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.239478 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.239567 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.239900 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.240007 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:45:19.241607 24415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:19.243968 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.227491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54816 (local address 127.1.201.62:40391)
0928 14:45:19.227584 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.227587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.229693 (+ 2106us) server_negotiation.cc:407] Connection header received
0928 14:45:19.229902 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.229905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.229946 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.229992 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.230919 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.231687 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.237981 (+ 6294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.238310 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.238418 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.240522 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.240529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.240535 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.240562 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.242247 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.242672 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.242675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.242677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.242769 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.243081 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.243084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.243086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.243305 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.243425 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.243716 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.243827 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:45:19.245112 24298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec: No bootstrap required, opened a new log
I20260928 14:45:19.247973 24348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [term 1 FOLLOWER]: Refusing update from remote peer 7a1bd4ade6a649ccb4080f548cd3eab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:19.248713 24382 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:19.249899 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:19.229235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.1.201.60:41557)
0928 14:45:19.229330 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.229334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.234019 (+ 4685us) server_negotiation.cc:407] Connection header received
0928 14:45:19.234121 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.234124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.234166 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.234231 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.242532 (+ 8301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.243214 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.244256 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:19.244572 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:19.244649 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:19.246275 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:19.246278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:19.246281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:19.246311 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:19.247874 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.248306 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.248309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.248311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.248389 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:19.248673 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:19.248676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:19.248678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:19.248876 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:19.249005 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:19.249569 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 14:45:19.249673 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:45:19.253319 24395 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a1bd4ade6a649ccb4080f548cd3eab3. Latest consensus state: current_term: 1 leader_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } } }
I20260928 14:45:19.253430 24395 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:19.254802 24382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } } }
I20260928 14:45:19.254910 24382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:19.256093 24395 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } } }
I20260928 14:45:19.256197 24395 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:19.256399 24298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } }
I20260928 14:45:19.256623 24298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26be03f4762c42b6b47d10fa8a245bec, State: Initialized, Role: FOLLOWER
I20260928 14:45:19.256961 24298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } }
I20260928 14:45:19.257913 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } } }
I20260928 14:45:19.258208 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:19.258432 24298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:19.258493 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "26be03f4762c42b6b47d10fa8a245bec"
I20260928 14:45:19.258687 24272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:19.260097 24272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a1bd4ade6a649ccb4080f548cd3eab3 in term 1.
I20260928 14:45:19.260630 24271 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [term 1 FOLLOWER]: Refusing update from remote peer 7a1bd4ade6a649ccb4080f548cd3eab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:19.262034 24382 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:19.263856 24415 catalog_manager.cc:1347] Generated new cluster ID: d07deaffb05c4a9b980524c62a6ca71d
I20260928 14:45:19.263911 24415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:19.269865 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a1bd4ade6a649ccb4080f548cd3eab3. Latest consensus state: current_term: 1 leader_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } } }
I20260928 14:45:19.270042 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:19.270167 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } } }
I20260928 14:45:19.270288 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:45:19.271560 24434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:45:19.271647 24434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:19.273464 24437 mvcc.cc:205] Tried to move back new op lower bound from 7334325122032078848 to 7334325121946140672. Current Snapshot: MvccSnapshot[applied={T|T < 7334325122032078848}]
I20260928 14:45:19.321969 24382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1bd4ade6a649ccb4080f548cd3eab3" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 40391 } } peers { permanent_uuid: "26be03f4762c42b6b47d10fa8a245bec" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 44861 } } peers { permanent_uuid: "a91bebfb956540369d44a7ea349e7e79" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 41557 } } }
I20260928 14:45:19.322119 24382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:19.326018 24415 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:19.327322 24415 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:19.338330 24415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a1bd4ade6a649ccb4080f548cd3eab3: Generated new TSK 0
I20260928 14:45:19.339102 24415 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:45:19.368185 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.365617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.1.201.61:44861)
0928 14:45:19.365746 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.365751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.367100 (+ 1349us) server_negotiation.cc:407] Connection header received
0928 14:45:19.367296 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.367299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.367353 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.367418 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.368053 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45650: BlockingRecv error: recv got EOF from 127.0.0.1:45650 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 14:45:19.369617 24224 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.366438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.1.201.62:40391)
0928 14:45:19.366551 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.366555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.368674 (+ 2119us) server_negotiation.cc:407] Connection header received
0928 14:45:19.368923 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.368926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.368974 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.369026 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.369501 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54830: BlockingRecv error: recv got EOF from 127.0.0.1:54830 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 14:45:19.371618 24379 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.370259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.1.201.60:41557)
0928 14:45:19.370368 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.370373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.370712 (+ 339us) server_negotiation.cc:407] Connection header received
0928 14:45:19.370923 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.370927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.370977 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.371027 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.371505 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47276: BlockingRecv error: recv got EOF from 127.0.0.1:47276 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 14:45:19.375914 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.374419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.1.201.61:44861)
0928 14:45:19.374514 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.374518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.374965 (+ 447us) server_negotiation.cc:407] Connection header received
0928 14:45:19.375151 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.375154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.375232 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.375291 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.375827 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45654: BlockingRecv error: recv got EOF from 127.0.0.1:45654 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 14:45:19.380558 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.379125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.1.201.61:44861)
0928 14:45:19.379221 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.379225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.379713 (+ 488us) server_negotiation.cc:407] Connection header received
0928 14:45:19.379887 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.379890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.379940 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.379990 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.380464 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45670: BlockingRecv error: recv got EOF from 127.0.0.1:45670 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 14:45:19.385979 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.384371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.1.201.61:44861)
0928 14:45:19.384467 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.384471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.384987 (+ 516us) server_negotiation.cc:407] Connection header received
0928 14:45:19.385224 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.385227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.385275 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.385326 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.385895 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45684: BlockingRecv error: recv got EOF from 127.0.0.1:45684 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 14:45:19.393347 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.391850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.1.201.61:44861)
0928 14:45:19.391988 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.391992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.392491 (+ 499us) server_negotiation.cc:407] Connection header received
0928 14:45:19.392658 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.392661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.392723 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.392787 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.393258 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45692: BlockingRecv error: recv got EOF from 127.0.0.1:45692 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 14:45:19.404536 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.403186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.1.201.61:44861)
0928 14:45:19.403275 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.403279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.403759 (+ 480us) server_negotiation.cc:407] Connection header received
0928 14:45:19.403920 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.403923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.403965 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.404010 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.404443 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45706: BlockingRecv error: recv got EOF from 127.0.0.1:45706 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 14:45:19.424204 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.422858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.1.201.61:44861)
0928 14:45:19.422959 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.422963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.423361 (+ 398us) server_negotiation.cc:407] Connection header received
0928 14:45:19.423521 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.423524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.423574 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.423669 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.424097 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45718: BlockingRecv error: recv got EOF from 127.0.0.1:45718 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 14:45:19.459822 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.458510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.1.201.61:44861)
0928 14:45:19.458643 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.458649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.458901 (+ 252us) server_negotiation.cc:407] Connection header received
0928 14:45:19.459069 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.459073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.459128 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.459226 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.459723 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45734: BlockingRecv error: recv got EOF from 127.0.0.1:45734 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 14:45:19.527179 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.526209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45746 (local address 127.1.201.61:44861)
0928 14:45:19.526332 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.526336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.526478 (+ 142us) server_negotiation.cc:407] Connection header received
0928 14:45:19.526580 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.526583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.526628 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.526725 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.527089 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45746: BlockingRecv error: recv got EOF from 127.0.0.1:45746 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 14:45:19.658680 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.657455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.1.201.61:44861)
0928 14:45:19.657606 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.657611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.657750 (+ 139us) server_negotiation.cc:407] Connection header received
0928 14:45:19.657890 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.657893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.657995 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.658077 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.658530 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45748: BlockingRecv error: recv got EOF from 127.0.0.1:45748 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 14:45:19.917621 24301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:19.916603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.1.201.61:44861)
0928 14:45:19.916769 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:19.916773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:19.916870 (+ 97us) server_negotiation.cc:407] Connection header received
0928 14:45:19.916991 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:19.916993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:19.917037 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:19.917109 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:19.917493 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45756: BlockingRecv error: recv got EOF from 127.0.0.1:45756 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:45:20.227396 24407 catalog_manager.cc:1259] Loaded cluster ID: d07deaffb05c4a9b980524c62a6ca71d
I20260928 14:45:20.227483 24407 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79: loading cluster ID for follower catalog manager: success
I20260928 14:45:20.229076 24407 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79: acquiring CA information for follower catalog manager: success
I20260928 14:45:20.230234 24407 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a91bebfb956540369d44a7ea349e7e79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:45:20.273006 24434 catalog_manager.cc:1259] Loaded cluster ID: d07deaffb05c4a9b980524c62a6ca71d
I20260928 14:45:20.273135 24434 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec: loading cluster ID for follower catalog manager: success
I20260928 14:45:20.274760 24434 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec: acquiring CA information for follower catalog manager: success
I20260928 14:45:20.276036 24434 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 26be03f4762c42b6b47d10fa8a245bec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:45:20.436985 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:20.432310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.1.201.61:44861)
0928 14:45:20.432583 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:20.432586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:20.432601 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:45:20.432765 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:20.432768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:20.432970 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:20.433084 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:20.433343 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:20.434216 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:20.435462 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:20.435786 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:20.435872 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:20.436126 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:20.436558 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:20.436648 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:20.436708 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 14:45:20.436755 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260928 14:45:20.443555 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:20.440129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.1.201.60:41557)
0928 14:45:20.440340 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:20.440344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:20.440356 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:45:20.440433 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:20.440436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:20.440547 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:20.440644 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:20.440846 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:20.441354 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:20.442210 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:20.442427 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:20.442506 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:20.442751 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:20.443180 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:20.443268 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:20.443316 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 14:45:20.443362 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260928 14:45:20.450706 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:20.446550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.1.201.61:44861)
0928 14:45:20.446674 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:20.446679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:20.446927 (+ 248us) server_negotiation.cc:407] Connection header received
0928 14:45:20.447127 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:20.447130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:20.447185 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:20.447252 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:20.447574 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:20.448294 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:20.449590 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:20.449898 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:20.450010 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:20.450256 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:20.450332 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:20.450413 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:20.450512 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 14:45:20.450565 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:45:20.451090 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:20.446918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.1.201.60:41557)
0928 14:45:20.447025 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:20.447031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:20.447066 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:45:20.447212 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:20.447218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:20.447269 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:20.447332 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:20.447622 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:20.448228 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:20.449577 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:20.449953 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:20.450058 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:20.450470 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:20.450541 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:20.450601 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:20.450884 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 14:45:20.450941 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:45:20.451432 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:20.446511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.1.201.62:40391)
0928 14:45:20.447101 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:20.447106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:20.447331 (+ 225us) server_negotiation.cc:407] Connection header received
0928 14:45:20.447538 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:20.447541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:20.447701 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:20.447826 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:20.448067 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:20.448690 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:20.449629 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:20.449993 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:20.450082 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:20.450526 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:20.450861 (+ 335us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:20.450942 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:20.451231 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 14:45:20.451288 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":66,"threads_started":1}
I20260928 14:45:20.461342 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 24153
I20260928 14:45:20.467635 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 24229
I20260928 14:45:20.473660 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 24305
2026-09-28T14:45:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3040 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:20 dist-test-slave-0n27 krb5kdc[24523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:20 dist-test-slave-0n27 krb5kdc[24523](info): set up 2 sockets
Sep 28 14:45:20 dist-test-slave-0n27 krb5kdc[24523](info): commencing operation
krb5kdc: starting...
W20260928 14:45:21.503089 1828 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/test-user.keytab.
Sep 28 14:45:21 dist-test-slave-0n27 krb5kdc[24523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.201.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
Entry for principal kudu/127.1.201.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab.
I20260928 14:45:21.624259 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:43781
--webserver_interface=127.1.201.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:43781,127.1.201.61:32811,127.1.201.60:36807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:21.733983 24536 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:45:21.734345 24536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:21.734421 24536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:21.734500 24536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:21.738514 24536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:21.738603 24536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:21.738641 24536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:21.738679 24536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:21.738714 24536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:21.743865 24536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:43781,127.1.201.61:32811,127.1.201.60:36807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.62.keytab
--principal=kudu/127.1.201.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.62:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.201.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:21.744931 24536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:21.745971 24536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:21.754417 24541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:21.754402 24542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:21.754402 24544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:21.754487 24536 server_base.cc:1034] running on GCE node
I20260928 14:45:21.755240 24536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:21.756307 24536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:21.757498 24536 hybrid_clock.cc:648] HybridClock initialized: now 1790606721757468 us; error 46 us; skew 500 ppm
Sep 28 14:45:21 dist-test-slave-0n27 krb5kdc[24523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:21.760175 24536 init.cc:377] Logged in from keytab as kudu/127.1.201.62@KRBTEST.COM (short username kudu)
I20260928 14:45:21.760998 24536 webserver.cc:460] Webserver started at http://127.1.201.62:38113/ using document root <none> and password file <none>
I20260928 14:45:21.761550 24536 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:21.761641 24536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:21.761929 24536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:21.763967 24536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/data/instance:
uuid: "fdcfeeb5ece349b7b454ca7e6d70d631"
format_stamp: "Formatted at 2026-09-28 14:45:21 on dist-test-slave-0n27"
server_key: "143f240b2d407317d05b6c4659b35fbc"
server_key_iv: "216a8b60a65bb06d0e978efc5b6bb5ac"
server_key_version: "encryptionkey@0"
I20260928 14:45:21.764536 24536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/wal/instance:
uuid: "fdcfeeb5ece349b7b454ca7e6d70d631"
format_stamp: "Formatted at 2026-09-28 14:45:21 on dist-test-slave-0n27"
server_key: "143f240b2d407317d05b6c4659b35fbc"
server_key_iv: "216a8b60a65bb06d0e978efc5b6bb5ac"
server_key_version: "encryptionkey@0"
I20260928 14:45:21.768251 24536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:45:21.770493 24551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:21.771690 24536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260928 14:45:21.771814 24536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/wal
uuid: "fdcfeeb5ece349b7b454ca7e6d70d631"
format_stamp: "Formatted at 2026-09-28 14:45:21 on dist-test-slave-0n27"
server_key: "143f240b2d407317d05b6c4659b35fbc"
server_key_iv: "216a8b60a65bb06d0e978efc5b6bb5ac"
server_key_version: "encryptionkey@0"
I20260928 14:45:21.771914 24536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:21.811582 24536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:21.814404 24536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:21.814604 24536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:21.821720 24536 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.62:43781
I20260928 14:45:21.822059 24603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.62:43781 every 8 connection(s)
I20260928 14:45:21.822714 24536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:21.825358 24604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:21.828954 24604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:21.830616 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 24536
I20260928 14:45:21.830806 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-0/wal/instance
I20260928 14:45:21.831120 1828 external_mini_cluster.cc:1428] Setting key 3e150e21076a593dfa71466c73997596
WARNING: no policy specified for kudu/127.1.201.61@KRBTEST.COM; defaulting to no policy
Sep 28 14:45:21 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.61@KRBTEST.COM" created.
I20260928 14:45:21.845387 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:21.831257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:43781 (local address 127.0.0.1:34994)
0928 14:45:21.831818 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:21.831878 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:21.833103 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:21.833971 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:21.833976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:21.834467 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:21.834851 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:21.834873 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:21.836103 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:21.836107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:21.837444 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:21.837460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:21.838173 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:21.838177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:21.838316 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:21.838884 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:21.838910 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:21.840478 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:21.842369 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:21.842378 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:21.842385 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:21.842710 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:21.843143 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:21.843146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:21.843148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:21.843296 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:21.843858 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:21.843868 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:21.844036 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 14:45:21.844216 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 14:45:21.844651 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":122,"thread_start_us":143,"threads_started":1}
I20260928 14:45:21.845391 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:21.831257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.1.201.62:43781)
0928 14:45:21.831805 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:21.831823 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:21.831995 (+ 172us) server_negotiation.cc:407] Connection header received
0928 14:45:21.833418 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:21.833425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:21.833701 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:21.833952 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:21.835016 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:21.835963 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:21.837573 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:21.837998 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:21.838122 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:21.840624 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:21.840629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:21.840639 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:21.840658 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:21.842270 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:21.842871 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:21.842880 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:21.842885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:21.843038 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:21.843386 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:21.843388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:21.843389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:21.843735 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:21.843829 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:21.844318 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 14:45:21.844586 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":429,"spinlock_wait_cycles":55168,"thread_start_us":158,"threads_started":1}
I20260928 14:45:21.847790 24604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:21.848448 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:21.848217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:32811 (local address 127.0.0.1:42086)
0928 14:45:21.848325 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:21.848396 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:45:21.848717 24555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.201.61:32811: connect: Connection refused (error 111)
I20260928 14:45:21.849581 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:21.849396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:32811 (local address 127.0.0.1:42090)
0928 14:45:21.849493 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:21.849534 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:45:21.849830 24604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.61:32811: Network error: Client connection negotiation failed: client connection to 127.1.201.61:32811: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab.
Entry for principal kudu/127.1.201.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab.
I20260928 14:45:21.859990 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.61:32811
--webserver_interface=127.1.201.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab
--principal=kudu/127.1.201.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:43781,127.1.201.61:32811,127.1.201.60:36807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:45:21.899078 24604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } attempt: 1
I20260928 14:45:21.899958 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:21.899667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:32811 (local address 127.0.0.1:42098)
0928 14:45:21.899822 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:21.899882 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 14:45:21.900648 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:21.900443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:32811 (local address 127.0.0.1:42100)
0928 14:45:21.900545 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:21.900595 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 14:45:21.900772 24604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.61:32811: Network error: Client connection negotiation failed: client connection to 127.1.201.61:32811: connect: Connection refused (error 111)
W20260928 14:45:21.968421 24611 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:45:21.968743 24611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:21.968806 24611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:21.968879 24611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:45:21.968920 24604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } attempt: 2
I20260928 14:45:21.969676 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:21.969395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:32811 (local address 127.0.0.1:42116)
0928 14:45:21.969533 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:21.969596 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:45:21.970371 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:21.970160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:32811 (local address 127.0.0.1:42128)
0928 14:45:21.970263 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:21.970308 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.61:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:45:21.970495 24604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.61:32811: Network error: Client connection negotiation failed: client connection to 127.1.201.61:32811: connect: Connection refused (error 111)
W20260928 14:45:21.973114 24611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:21.973196 24611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:21.973233 24611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:21.973268 24611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:21.973302 24611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:21.978440 24611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:43781,127.1.201.61:32811,127.1.201.60:36807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.61.keytab
--principal=kudu/127.1.201.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.61:32811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.201.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:21.979440 24611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:21.980396 24611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:21.988715 24617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:21.988842 24611 server_base.cc:1034] running on GCE node
W20260928 14:45:21.988726 24616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:21.988749 24619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:21.989512 24611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:21.990604 24611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:21.991797 24611 hybrid_clock.cc:648] HybridClock initialized: now 1790606721991784 us; error 31 us; skew 500 ppm
Sep 28 14:45:21 dist-test-slave-0n27 krb5kdc[24523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:21.994477 24611 init.cc:377] Logged in from keytab as kudu/127.1.201.61@KRBTEST.COM (short username kudu)
I20260928 14:45:21.995301 24611 webserver.cc:460] Webserver started at http://127.1.201.61:40227/ using document root <none> and password file <none>
I20260928 14:45:21.995851 24611 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:21.995911 24611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:21.996064 24611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:21.998005 24611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/data/instance:
uuid: "7b46a1868274494cb231f5b6d5c77e2b"
format_stamp: "Formatted at 2026-09-28 14:45:21 on dist-test-slave-0n27"
server_key: "ce359db261fb290c62117d5648648340"
server_key_iv: "336f99a170fe3baf23485425c4bb46af"
server_key_version: "encryptionkey@0"
I20260928 14:45:21.998499 24611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/wal/instance:
uuid: "7b46a1868274494cb231f5b6d5c77e2b"
format_stamp: "Formatted at 2026-09-28 14:45:21 on dist-test-slave-0n27"
server_key: "ce359db261fb290c62117d5648648340"
server_key_iv: "336f99a170fe3baf23485425c4bb46af"
server_key_version: "encryptionkey@0"
I20260928 14:45:22.001940 24611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:45:22.004231 24626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:22.005391 24611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:45:22.005492 24611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/wal
uuid: "7b46a1868274494cb231f5b6d5c77e2b"
format_stamp: "Formatted at 2026-09-28 14:45:21 on dist-test-slave-0n27"
server_key: "ce359db261fb290c62117d5648648340"
server_key_iv: "336f99a170fe3baf23485425c4bb46af"
server_key_version: "encryptionkey@0"
I20260928 14:45:22.005601 24611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:22.017196 24611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:22.019886 24611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:22.020100 24611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:22.026654 24611 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.61:32811
I20260928 14:45:22.026926 24678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.61:32811 every 8 connection(s)
I20260928 14:45:22.027575 24611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/data/info.pb
I20260928 14:45:22.030169 24679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:22.033674 24679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:22.036157 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 24611
I20260928 14:45:22.036332 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-1/wal/instance
I20260928 14:45:22.036652 1828 external_mini_cluster.cc:1428] Setting key e41fb7984bd10326483b577c624ea96a
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.201.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.201.60@KRBTEST.COM" created.
I20260928 14:45:22.047912 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.035529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.1.201.62:43781)
0928 14:45:22.035717 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.035720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.036443 (+ 723us) server_negotiation.cc:407] Connection header received
0928 14:45:22.037417 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.037420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.037479 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.037580 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.038710 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.039223 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.040326 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.040557 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.040665 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.043496 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.043499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.043501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.043525 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.045556 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.046113 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.046116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.046117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.046219 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.046557 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.046560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.046561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.046734 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.046846 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.047497 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.047649 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 14:45:22.048471 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.035849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:43781 (local address 127.0.0.1:34998)
0928 14:45:22.036299 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.036349 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:22.037222 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:22.037739 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:22.037746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:22.038133 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:22.038563 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.038576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.039350 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.039353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.040183 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.040191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.040701 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.040705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.040876 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.041474 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:22.041490 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:22.043328 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:22.045689 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.045694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.045704 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.046020 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.046340 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.046343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.046344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.046472 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.046868 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:22.046875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:22.047212 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 14:45:22.047387 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 14:45:22.047626 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":124,"threads_started":1}
I20260928 14:45:22.050022 24679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
I20260928 14:45:22.059288 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.050515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:32811 (local address 127.0.0.1:42142)
0928 14:45:22.050629 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.050644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:22.050719 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:22.051309 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:22.051315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:22.051517 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:22.051648 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.051658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.052590 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.052593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.053133 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.053141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.054024 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.054027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.054097 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.054533 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:22.054543 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:22.055878 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:22.057674 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.057676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.057678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.057917 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.058321 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.058323 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.058325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.058395 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.058912 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:22.058917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:22.059000 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 14:45:22.059076 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:45:22.059167 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:45:22.059525 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.050654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.1.201.61:32811)
0928 14:45:22.050958 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.050966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.050981 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:45:22.051011 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.051015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.051154 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.051270 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.051771 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.052485 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.053249 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.053918 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.054031 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.056009 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.056018 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.056032 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.056055 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.057551 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.058097 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.058105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.058108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.058211 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.058492 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.058494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.058495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.058798 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.058911 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.059144 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.059256 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":140,"threads_started":1}
I20260928 14:45:22.061322 24679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:22.061657 24604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } attempt: 3
I20260928 14:45:22.062063 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.061746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46562)
0928 14:45:22.061920 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.062000 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260928 14:45:22.062338 24627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
I20260928 14:45:22.063265 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.063022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46578)
0928 14:45:22.063153 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.063210 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 14:45:22.063432 24679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:36807: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.201.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab.
Entry for principal kudu/127.1.201.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab.
I20260928 14:45:22.064500 1828 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskScXicz/build/debug/bin/kudu
/tmp/dist-test-taskScXicz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.60:36807
--webserver_interface=127.1.201.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab
--principal=kudu/127.1.201.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.201.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.201.62:43781,127.1.201.61:32811,127.1.201.60:36807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
I20260928 14:45:22.070700 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.062178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:32811 (local address 127.0.0.1:42154)
0928 14:45:22.062312 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.062329 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:22.062411 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:22.063012 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:22.063017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:22.063197 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:22.063290 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.063295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.064298 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.064301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.064945 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.064954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.065472 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.065476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.065583 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.066075 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:22.066094 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:22.067359 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:22.069244 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.069246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.069248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.069431 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.069733 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.069735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.069736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.069877 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.070326 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:22.070328 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:22.070406 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 14:45:22.070481 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:45:22.070579 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:45:22.070916 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.062202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.1.201.61:32811)
0928 14:45:22.062824 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.062827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.062837 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:45:22.062865 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.062867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.062898 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.062990 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.063441 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.064184 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.065058 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.065362 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.065471 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.067484 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.067491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.067497 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.067526 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.069136 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.069529 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.069535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.069539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.069631 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.069984 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.069989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.069993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.070230 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.070344 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.070570 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.070697 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566}
I20260928 14:45:22.072073 24604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:45:22.072773 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.072554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46584)
0928 14:45:22.072666 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.072721 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 14:45:22.073525 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.073307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46588)
0928 14:45:22.073445 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.073483 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260928 14:45:22.073715 24604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:36807: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
I20260928 14:45:22.104887 24604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } attempt: 1
I20260928 14:45:22.105718 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.105429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46596)
0928 14:45:22.105577 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.105634 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 14:45:22.106446 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.106234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46612)
0928 14:45:22.106339 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.106388 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:45:22.106575 24604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:36807: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
I20260928 14:45:22.112684 24679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } attempt: 1
I20260928 14:45:22.113349 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.113110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46614)
0928 14:45:22.113238 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.113288 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 14:45:22.114035 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.113838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46626)
0928 14:45:22.113941 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.113988 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 14:45:22.114156 24679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:36807: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
W20260928 14:45:22.175098 24686 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:45:22.175412 24686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:22.175470 24686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:22.175542 24686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:22.180083 24686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:22.180156 24686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:22.180192 24686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:22.180228 24686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:22.180270 24686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:22.181728 24604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } attempt: 2
I20260928 14:45:22.182312 24679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } attempt: 2
I20260928 14:45:22.182720 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.182285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46638)
0928 14:45:22.182517 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.182607 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 14:45:22.183079 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.182708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46646)
0928 14:45:22.182924 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.183001 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 14:45:22.183815 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.183548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46650)
0928 14:45:22.183672 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.183750 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:45:22.183848 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.183568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46664)
0928 14:45:22.183712 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.183768 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 14:45:22.183997 24604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:36807: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
W20260928 14:45:22.183995 24679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.201.60:36807: Network error: Client connection negotiation failed: client connection to 127.1.201.60:36807: connect: Connection refused (error 111)
I20260928 14:45:22.186363 24686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.201.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.201.62:43781,127.1.201.61:32811,127.1.201.60:36807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/krb5kdc/kudu_127.1.201.60.keytab
--principal=kudu/127.1.201.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.201.60:36807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.201.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:22.187363 24686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:22.188385 24686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:22.196664 24692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:22.196664 24691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:22.196800 24686 server_base.cc:1034] running on GCE node
W20260928 14:45:22.196964 24694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:22.197492 24686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:22.198570 24686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:22.199736 24686 hybrid_clock.cc:648] HybridClock initialized: now 1790606722199725 us; error 33 us; skew 500 ppm
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:22.202453 24686 init.cc:377] Logged in from keytab as kudu/127.1.201.60@KRBTEST.COM (short username kudu)
I20260928 14:45:22.203332 24686 webserver.cc:460] Webserver started at http://127.1.201.60:42207/ using document root <none> and password file <none>
I20260928 14:45:22.203900 24686 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:22.203989 24686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:22.204219 24686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:22.206257 24686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/data/instance:
uuid: "f21116d4ade0420eaba673195b8748cb"
format_stamp: "Formatted at 2026-09-28 14:45:22 on dist-test-slave-0n27"
server_key: "9ab4448c13782e05498d0acc6566c61b"
server_key_iv: "1a9b8b63f97f7af49982c7f5678d773c"
server_key_version: "encryptionkey@0"
I20260928 14:45:22.206808 24686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/wal/instance:
uuid: "f21116d4ade0420eaba673195b8748cb"
format_stamp: "Formatted at 2026-09-28 14:45:22 on dist-test-slave-0n27"
server_key: "9ab4448c13782e05498d0acc6566c61b"
server_key_iv: "1a9b8b63f97f7af49982c7f5678d773c"
server_key_version: "encryptionkey@0"
I20260928 14:45:22.210584 24686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:45:22.212895 24701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:22.214151 24686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:45:22.214293 24686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/data,/tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/wal
uuid: "f21116d4ade0420eaba673195b8748cb"
format_stamp: "Formatted at 2026-09-28 14:45:22 on dist-test-slave-0n27"
server_key: "9ab4448c13782e05498d0acc6566c61b"
server_key_iv: "1a9b8b63f97f7af49982c7f5678d773c"
server_key_version: "encryptionkey@0"
I20260928 14:45:22.214398 24686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:22.239012 24686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:22.242262 24686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:22.242470 24686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:22.249042 24686 rpc_server.cc:307] RPC server started. Bound to: 127.1.201.60:36807
I20260928 14:45:22.249316 24753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.201.60:36807 every 8 connection(s)
I20260928 14:45:22.249989 24686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/data/info.pb
I20260928 14:45:22.250914 1828 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskScXicz/build/debug/bin/kudu as pid 24686
I20260928 14:45:22.251049 1828 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskScXicz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606444849692-1828-0/minicluster-data/master-2/wal/instance
I20260928 14:45:22.251412 1828 external_mini_cluster.cc:1428] Setting key b09e6ea63952042f63a720e64f4cec31
I20260928 14:45:22.253188 24754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:22.257416 24754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
I20260928 14:45:22.265305 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.252654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46670 (local address 127.1.201.60:36807)
0928 14:45:22.253026 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.253051 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.253084 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:45:22.253739 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.253756 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.254144 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.254415 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.255257 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.256293 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.257407 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.258003 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.258173 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.260861 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.260866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.260875 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.260901 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.262546 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.262969 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.262978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.262982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.263083 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.263357 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.263360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.263362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.263744 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.263935 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.264197 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.264462 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":123,"threads_started":1}
I20260928 14:45:22.268556 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.259411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.1.201.62:43781)
0928 14:45:22.259562 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.259566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.259905 (+ 339us) server_negotiation.cc:407] Connection header received
0928 14:45:22.260064 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.260067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.260117 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.260201 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.260892 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.261383 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.262290 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.262516 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.262620 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.264739 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.264742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.264744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.264767 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.266720 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.267227 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.267230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.267232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.267339 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.267669 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.267673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.267675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.267905 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.267997 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.268364 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.268455 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:45:22.268679 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.259343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.62:43781 (local address 127.0.0.1:35012)
0928 14:45:22.259776 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.259829 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:22.259958 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:22.260249 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:22.260254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:22.260637 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:22.260771 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.260777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.261502 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.261505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.262173 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.262182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.262623 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.262627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.262745 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.263230 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:22.263241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:22.264616 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:22.266838 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.266846 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.266851 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.267134 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.267449 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.267452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.267454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.267547 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.268006 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:22.268012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:22.268141 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 14:45:22.268294 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 14:45:22.268472 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":137,"threads_started":1}
I20260928 14:45:22.269809 24754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.62@KRBTEST.COM
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
I20260928 14:45:22.275328 24679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } attempt: 3
I20260928 14:45:22.276901 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.267435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.1.201.62:43781)
0928 14:45:22.267777 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.267785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.267797 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:45:22.267826 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.267829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.267864 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.267954 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.268684 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.269391 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.270486 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.270819 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.270922 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.273121 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.273124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.273126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.273146 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.274594 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.275058 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.275060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.275061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.275164 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.275511 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.275515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.275520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.275746 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.275958 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.276667 (+ 709us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.276778 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260928 14:45:22.277612 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.270243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.61:32811 (local address 127.0.0.1:42168)
0928 14:45:22.270390 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.270400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:22.270451 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:22.270708 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:22.270710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:22.270827 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:22.270905 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.270909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.271630 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.271636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.272127 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.272134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.272592 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.272595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.272697 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.273150 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:22.273161 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:22.274243 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:22.276076 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.276079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.276080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.276327 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.276638 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.276641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.276643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.276746 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.277180 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:22.277183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:22.277260 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 14:45:22.277351 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:45:22.277464 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 14:45:22.277979 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.270383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.1.201.61:32811)
0928 14:45:22.270506 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.270510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.270522 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:45:22.270561 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.270564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.270604 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.270681 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.271022 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.271526 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.272254 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.272489 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.272591 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.274379 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.274382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.274385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.274410 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.275961 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.276438 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.276441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.276443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.276522 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.276849 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.276852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.276854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.277048 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.277113 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.277421 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.277550 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:45:22.278631 24754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.61@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
I20260928 14:45:22.283129 24604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } attempt: 3
I20260928 14:45:22.284569 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.275706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46676)
0928 14:45:22.275867 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.275881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:22.275966 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:22.276234 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:22.276237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:22.276438 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:22.276533 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.276540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.277509 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.277515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.278346 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.278355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.278904 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.278907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.278995 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.279584 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:22.279598 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:22.280980 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:22.282957 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.282960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.282962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.283222 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.283510 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.283513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.283515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.283605 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.284052 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:22.284055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:22.284146 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 14:45:22.284238 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:45:22.284386 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.60@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
I20260928 14:45:22.285408 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.275763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46676 (local address 127.1.201.60:36807)
0928 14:45:22.275883 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.275887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.276068 (+ 181us) server_negotiation.cc:407] Connection header received
0928 14:45:22.276106 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.276108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.276138 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.276219 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.276690 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.277405 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.278452 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.278780 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.278890 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.281116 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.281120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.281123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.281146 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.282844 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.283323 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.283325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.283327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.283402 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.283722 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.283725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.283727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.283933 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.285091 (+ 1158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.285167 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.285277 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.201.61@KRBTEST.COM
I20260928 14:45:22.289021 24679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b: Bootstrap starting.
I20260928 14:45:22.289218 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.279584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46684)
0928 14:45:22.279739 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.279752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:22.279813 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:22.280495 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:22.280499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:22.280670 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:22.280757 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.280764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.281682 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.281688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.282504 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.282514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.283253 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.283259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.283344 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.283869 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:22.283881 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:22.285756 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:22.287651 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.287653 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.287655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.287866 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.288286 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.288289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.288291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.288395 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.288817 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:22.288820 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:22.288890 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 14:45:22.288977 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:45:22.289084 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
Sep 28 14:45:22 dist-test-slave-0n27 krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.201.62@KRBTEST.COM for kudu/127.1.201.60@KRBTEST.COM
I20260928 14:45:22.289604 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.279606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.1.201.60:36807)
0928 14:45:22.280311 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.280318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.280327 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:45:22.280352 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.280354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.280382 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.280447 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.280876 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.281592 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.282619 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.283021 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.283117 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.285901 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.285905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.285907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.285927 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.287542 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.287980 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.287985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.287989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.288082 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.288502 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.288504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.288506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.288704 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.288788 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.289368 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.289463 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":96,"threads_started":1}
I20260928 14:45:22.290995 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.279049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.1.201.61:32811)
0928 14:45:22.279156 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.279159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.279237 (+ 78us) server_negotiation.cc:407] Connection header received
0928 14:45:22.279366 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.279369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.279411 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.279490 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.280728 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.281372 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.282487 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.282747 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.282853 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.287074 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.287077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.287080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.287105 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.288885 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.289375 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.289380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.289384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.289462 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.289763 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.289766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.289768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.290001 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.290699 (+ 698us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.290776 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.290873 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:45:22.291815 24679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:22.292544 24754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb: Bootstrap starting.
I20260928 14:45:22.292826 24679 log.cc:824] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:22.294423 1828 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:22.295154 24754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:22.295264 24679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b: No bootstrap required, opened a new log
I20260928 14:45:22.295974 24754 log.cc:824] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:22.297196 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.283529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.201.60:36807 (local address 127.0.0.1:46690)
0928 14:45:22.283674 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:22.283688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:22.283754 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:22.284314 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:22.284319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:22.284535 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:22.284642 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.284647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.285632 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.285637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.286432 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:22.286442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.287285 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.287288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:22.287639 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.288284 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:22.288301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:22.289678 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:22.295555 (+ 5877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.295558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.295560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.295832 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.296192 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:22.296195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:22.296197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:22.296288 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:22.296709 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:22.296712 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:22.296808 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 14:45:22.296895 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:45:22.297038 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 14:45:22.295768 24578 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35022
I20260928 14:45:22.297705 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.283756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.1.201.60:36807)
0928 14:45:22.284047 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.284058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.284075 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:45:22.284115 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.284119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.284165 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.284760 (+ 595us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.284884 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.285426 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.286566 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.287156 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.288151 (+ 995us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.289842 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.289847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.289849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.289870 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.295423 (+ 5553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.295955 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.295958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.295960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.296074 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.296403 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.296406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.296408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.296596 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.297365 (+ 769us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.297442 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.297549 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
I20260928 14:45:22.297966 24754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb: No bootstrap required, opened a new log
I20260928 14:45:22.298992 24679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } }
I20260928 14:45:22.299151 24679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b46a1868274494cb231f5b6d5c77e2b, State: Initialized, Role: FOLLOWER
I20260928 14:45:22.299398 24679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } }
I20260928 14:45:22.299787 24764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } } }
I20260928 14:45:22.299902 24764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:22.300525 24679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:22.304800 24604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631: Bootstrap starting.
W20260928 14:45:22.306337 24775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:45:22.306411 24775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:22.307276 24604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:22.304330 24611 master_runner.cc:186] Error getting master registration for 127.1.201.62:43781: OK, instance_id { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" instance_seqno: 1790606721819766 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:45:22.307615 24611 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:45:22.308095 24604 log.cc:824] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:22.309405 24754 raft_consensus.cc:348] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } }
I20260928 14:45:22.309656 24754 raft_consensus.cc:729] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21116d4ade0420eaba673195b8748cb, State: Initialized, Role: FOLLOWER
I20260928 14:45:22.310037 24754 consensus_queue.cc:260] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } }
I20260928 14:45:22.310608 24778 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } } }
I20260928 14:45:22.310844 24778 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:22.311257 24604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631: No bootstrap required, opened a new log
I20260928 14:45:22.311352 24754 sys_catalog.cc:564] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:22.314355 24604 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } }
I20260928 14:45:22.314554 24604 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdcfeeb5ece349b7b454ca7e6d70d631, State: Initialized, Role: FOLLOWER
I20260928 14:45:22.314867 24604 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } }
W20260928 14:45:22.316068 24557 tablet.cc:2369] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:45:22.316169 24789 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:45:22.316222 24789 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:22.316079 24604 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:22.314855 24686 master_runner.cc:186] Error getting master registration for 127.1.201.62:43781: OK, instance_id { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" instance_seqno: 1790606721819766 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:45:22.317572 24686 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:45:22.317852 24578 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 14:45:22.317994 24578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:22.319235 24578 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } }
I20260928 14:45:22.320132 24578 leader_election.cc:288] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [CANDIDATE]: Term 1 election: Requested vote from peers 7b46a1868274494cb231f5b6d5c77e2b (127.1.201.61:32811), f21116d4ade0420eaba673195b8748cb (127.1.201.60:36807)
I20260928 14:45:22.320997 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7b46a1868274494cb231f5b6d5c77e2b"
I20260928 14:45:22.321245 24653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:22.322527 24653 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdcfeeb5ece349b7b454ca7e6d70d631 in term 1.
I20260928 14:45:22.322983 24555 leader_election.cc:302] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b46a1868274494cb231f5b6d5c77e2b, fdcfeeb5ece349b7b454ca7e6d70d631; no voters:
I20260928 14:45:22.323405 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } } }
I20260928 14:45:22.323495 24728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f21116d4ade0420eaba673195b8748cb"
I20260928 14:45:22.323581 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:22.323696 24728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:22.323763 24807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:22.323848 24807 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [term 1 LEADER]: Becoming Leader. State: Replica: fdcfeeb5ece349b7b454ca7e6d70d631, State: Running, Role: LEADER
I20260928 14:45:22.324113 24807 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } }
I20260928 14:45:22.324905 24728 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdcfeeb5ece349b7b454ca7e6d70d631 in term 1.
I20260928 14:45:22.325805 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdcfeeb5ece349b7b454ca7e6d70d631. Latest consensus state: current_term: 1 leader_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } } }
I20260928 14:45:22.325901 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:22.328622 24536 master_runner.cc:263] No leader master found from master fdcfeeb5ece349b7b454ca7e6d70d631
I20260928 14:45:22.328709 24536 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:45:22.335677 24809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:22.340490 24809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:22.344080 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.334354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.1.201.62:43781)
0928 14:45:22.334476 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.334483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.334720 (+ 237us) server_negotiation.cc:407] Connection header received
0928 14:45:22.334921 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.334924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.334980 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.335037 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.336273 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.337012 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.338132 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.338492 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.338566 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.340392 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.340396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.340398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.340424 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.342133 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.342631 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.342635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.342636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.342722 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.343052 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.343057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.343059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.343294 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.343390 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.343793 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.343923 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:45:22.352762 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.334854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.1.201.61:32811)
0928 14:45:22.334960 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.334965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.343069 (+ 8104us) server_negotiation.cc:407] Connection header received
0928 14:45:22.343250 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.343254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.343305 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.343357 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.344444 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.345166 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.346431 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.346771 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.346860 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.349017 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.349021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.349023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.349065 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.350740 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.351203 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.351207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.351209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.351310 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.351787 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.351790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.351792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.352005 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.352126 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.352471 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.352588 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:45:22.357054 24728 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [term 1 FOLLOWER]: Refusing update from remote peer fdcfeeb5ece349b7b454ca7e6d70d631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:22.357265 24653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [term 1 FOLLOWER]: Refusing update from remote peer fdcfeeb5ece349b7b454ca7e6d70d631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:22.357743 24791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:22.358299 24807 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:22.362821 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:22.335256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.1.201.60:36807)
0928 14:45:22.335382 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.335386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.345546 (+ 10160us) server_negotiation.cc:407] Connection header received
0928 14:45:22.345834 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.345838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.345910 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.345973 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.346882 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.347883 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.355330 (+ 7447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:22.355741 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:22.355822 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:22.358646 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:22.358650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:22.358653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:22.358683 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:22.360564 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.361253 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.361257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.361259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.361356 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:22.361677 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:22.361680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:22.361682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:22.361924 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:22.362041 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:22.362431 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 14:45:22.362590 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:45:22.364529 24764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdcfeeb5ece349b7b454ca7e6d70d631. Latest consensus state: current_term: 1 leader_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } } }
I20260928 14:45:22.364672 24764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:22.365342 24778 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdcfeeb5ece349b7b454ca7e6d70d631. Latest consensus state: current_term: 1 leader_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } } }
I20260928 14:45:22.365491 24778 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:22.378877 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } } }
I20260928 14:45:22.379009 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:22.379120 24814 mvcc.cc:205] Tried to move back new op lower bound from 7334325134753660928 to 7334325134642667520. Current Snapshot: MvccSnapshot[applied={T|T < 7334325134753660928}]
I20260928 14:45:22.380283 24764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } } }
I20260928 14:45:22.380391 24764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:22.382797 24791 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } } }
I20260928 14:45:22.382925 24791 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:22.384071 24778 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcfeeb5ece349b7b454ca7e6d70d631" member_type: VOTER last_known_addr { host: "127.1.201.62" port: 43781 } } peers { permanent_uuid: "7b46a1868274494cb231f5b6d5c77e2b" member_type: VOTER last_known_addr { host: "127.1.201.61" port: 32811 } } peers { permanent_uuid: "f21116d4ade0420eaba673195b8748cb" member_type: VOTER last_known_addr { host: "127.1.201.60" port: 36807 } } }
I20260928 14:45:22.384198 24778 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:45:22.388404 24816 mvcc.cc:205] Tried to move back new op lower bound from 7334325134753660928 to 7334325134642667520. Current Snapshot: MvccSnapshot[applied={T|T < 7334325134753660928}]
I20260928 14:45:22.391681 24809 catalog_manager.cc:1347] Generated new cluster ID: fe25c5ad005a498e92733510f90b1644
I20260928 14:45:22.391750 24809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:22.427066 24809 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:22.430450 24809 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:22.443399 24809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdcfeeb5ece349b7b454ca7e6d70d631: Generated new TSK 0
I20260928 14:45:22.444200 24809 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:45:22.476881 24759 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.474529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.1.201.62:43781)
0928 14:45:22.474669 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.474674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.475894 (+ 1220us) server_negotiation.cc:407] Connection header received
0928 14:45:22.476045 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.476048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.476097 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.476184 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.476754 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35034: BlockingRecv error: recv got EOF from 127.0.0.1:35034 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 14:45:22.477963 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.475198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.1.201.61:32811)
0928 14:45:22.475312 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.475316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.476987 (+ 1671us) server_negotiation.cc:407] Connection header received
0928 14:45:22.477162 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.477166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.477215 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.477284 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.477851 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42200: BlockingRecv error: recv got EOF from 127.0.0.1:42200 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 14:45:22.479131 24761 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.476358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.1.201.60:36807)
0928 14:45:22.476488 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.476493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.477339 (+ 846us) server_negotiation.cc:407] Connection header received
0928 14:45:22.478425 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.478429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.478481 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.478563 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.479024 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46710: BlockingRecv error: recv got EOF from 127.0.0.1:46710 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 14:45:22.483156 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.481817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.1.201.61:32811)
0928 14:45:22.481953 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.481959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.482243 (+ 284us) server_negotiation.cc:407] Connection header received
0928 14:45:22.482451 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.482454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.482520 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.482569 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.483052 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42202: BlockingRecv error: recv got EOF from 127.0.0.1:42202 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 14:45:22.487545 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.486252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.1.201.61:32811)
0928 14:45:22.486353 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.486357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.486800 (+ 443us) server_negotiation.cc:407] Connection header received
0928 14:45:22.486951 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.486954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.486996 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.487039 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.487462 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42208: BlockingRecv error: recv got EOF from 127.0.0.1:42208 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 14:45:22.492544 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.491184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.1.201.61:32811)
0928 14:45:22.491298 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.491302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.491763 (+ 461us) server_negotiation.cc:407] Connection header received
0928 14:45:22.491926 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.491928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.491970 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.492015 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.492463 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42210: BlockingRecv error: recv got EOF from 127.0.0.1:42210 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 14:45:22.499815 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.498313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.1.201.61:32811)
0928 14:45:22.498423 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.498427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.498922 (+ 495us) server_negotiation.cc:407] Connection header received
0928 14:45:22.499086 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.499089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.499133 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.499181 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.499702 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42218: BlockingRecv error: recv got EOF from 127.0.0.1:42218 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 14:45:22.511336 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.509890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.1.201.61:32811)
0928 14:45:22.509983 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.509987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.510447 (+ 460us) server_negotiation.cc:407] Connection header received
0928 14:45:22.510619 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.510622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.510667 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.510713 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.511238 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42234: BlockingRecv error: recv got EOF from 127.0.0.1:42234 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 14:45:22.530954 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.529678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.1.201.61:32811)
0928 14:45:22.529832 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.529836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.530099 (+ 263us) server_negotiation.cc:407] Connection header received
0928 14:45:22.530272 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.530276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.530332 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.530418 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.530847 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42246: BlockingRecv error: recv got EOF from 127.0.0.1:42246 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 14:45:22.565950 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.564771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42250 (local address 127.1.201.61:32811)
0928 14:45:22.564926 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.564930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.565053 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:45:22.565222 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.565226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.565281 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.565375 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.565833 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42250: BlockingRecv error: recv got EOF from 127.0.0.1:42250 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 14:45:22.633140 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.632022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.1.201.61:32811)
0928 14:45:22.632174 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.632177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.632325 (+ 148us) server_negotiation.cc:407] Connection header received
0928 14:45:22.632463 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.632466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.632515 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.632597 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.633038 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42260: BlockingRecv error: recv got EOF from 127.0.0.1:42260 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 14:45:22.764571 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:22.763281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.1.201.61:32811)
0928 14:45:22.763452 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:22.763458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:22.763641 (+ 183us) server_negotiation.cc:407] Connection header received
0928 14:45:22.763839 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:22.763842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:22.763894 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:22.763985 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:22.764467 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42274: BlockingRecv error: recv got EOF from 127.0.0.1:42274 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 14:45:23.023890 24684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:23.022889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.1.201.61:32811)
0928 14:45:23.023023 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:23.023027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:23.023128 (+ 101us) server_negotiation.cc:407] Connection header received
0928 14:45:23.023252 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:23.023254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:23.023302 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:23.023382 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:23.023792 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42290: BlockingRecv error: recv got EOF from 127.0.0.1:42290 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:45:23.308223 24775 catalog_manager.cc:1259] Loaded cluster ID: fe25c5ad005a498e92733510f90b1644
I20260928 14:45:23.308316 24775 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b: loading cluster ID for follower catalog manager: success
I20260928 14:45:23.310766 24775 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b: acquiring CA information for follower catalog manager: success
I20260928 14:45:23.312265 24775 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b46a1868274494cb231f5b6d5c77e2b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:45:23.317401 24789 catalog_manager.cc:1259] Loaded cluster ID: fe25c5ad005a498e92733510f90b1644
I20260928 14:45:23.317456 24789 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb: loading cluster ID for follower catalog manager: success
I20260928 14:45:23.319312 24789 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb: acquiring CA information for follower catalog manager: success
I20260928 14:45:23.320446 24789 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f21116d4ade0420eaba673195b8748cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:45:23.542065 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:23.538000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.1.201.61:32811)
0928 14:45:23.538280 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:23.538285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:23.538393 (+ 108us) server_negotiation.cc:407] Connection header received
0928 14:45:23.538569 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:23.538574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:23.538705 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:23.538823 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:23.539084 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:23.539596 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:23.540466 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:23.540692 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:23.540803 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:23.541123 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:23.541574 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:23.541714 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:23.541814 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:45:23.541871 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
I20260928 14:45:23.548312 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:23.544991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.1.201.60:36807)
0928 14:45:23.545231 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:23.545234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:23.545246 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:45:23.545318 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:23.545320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:23.545425 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:23.545515 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:23.545710 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:23.546284 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:23.547038 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:23.547239 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:23.547307 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:23.547541 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:23.547945 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:23.548028 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:23.548078 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 14:45:23.548122 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
I20260928 14:45:23.555138 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:23.551241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.1.201.60:36807)
0928 14:45:23.551357 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:23.551363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:23.551448 (+ 85us) server_negotiation.cc:407] Connection header received
0928 14:45:23.551638 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:23.551642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:23.551689 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:23.551763 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:23.552124 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:23.552813 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:23.554035 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:23.554354 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:23.554448 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:23.554715 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:23.554777 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:23.554857 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:23.554942 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 14:45:23.554988 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:45:23.555578 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:23.551691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.1.201.61:32811)
0928 14:45:23.551852 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:23.551856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:23.552207 (+ 351us) server_negotiation.cc:407] Connection header received
0928 14:45:23.552291 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:23.552294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:23.552340 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:23.552414 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:23.552680 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:23.553359 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:23.554416 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:23.554721 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:23.554795 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:23.555119 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:23.555183 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:23.555240 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:23.555406 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 14:45:23.555455 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:45:23.555755 24899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:23.551112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.1.201.62:43781)
0928 14:45:23.551363 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:23.551368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:23.551408 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:45:23.551601 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:23.551607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:23.551758 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:23.551869 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:23.552236 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:23.552981 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:23.554138 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:23.554363 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:23.554471 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:23.554847 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:23.555173 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:23.555259 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:23.555529 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 14:45:23.555586 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260928 14:45:23.559170 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 24536
I20260928 14:45:23.567183 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 24611
I20260928 14:45:23.575697 1828 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskScXicz/build/debug/bin/kudu with pid 24686
2026-09-28T14:45:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3102 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6142 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (278723 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST